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ABSTRACT 

As the second volume in a 4-volume evaluation 'report 
on the University of Massachusetts Non-Formal Education Project 
(UHass NFEP) in rural EcuadoCr this volume details the evaluation^^ 
design. Cited as basic to the evaluation design are questions which 
ask: (1) What kinds of effects (changes) can be observed? and (2> 
What are characteristics of the materials and procedures as they can 
be logically and/or empirically related to the desirable outcomes? 
Cited as major components of the design are: (1) The experitaental 
field implementation and concurrent evaluation of the UMass NFEP 
education games that were among the most widely used/accepted 
education. games; and (2) The UMass non-foraal education intervention 
impact study (an^impact profile^ a gualitati-ve analysis of 
intervention based on observations and interviews r and individual 
testing of game session participants) • Additionally^ this report 
describes: (1) 'Materials Selected for University of California at Los 
Angeles Field Implementation and Evaluation; (2) Instruments (a set 
of individually administered tests given in a pre-second-post basis 
to measure literacy^ numerical skillsr and critical cpnscioOsness or * 
attitudes); (3) Staffing (selection criteria^ training^ and 
procedures) ; (4) Implementation Procedures (selection of communities 
and control and experimental groups); (5) ^Criteria for Field 
Supervision and Quality Control. (JC) 



/ 



Documents acquired by ERIC include many informal unpublished materials not available from other sources. ERIC makes evei^ 
effort to obtain the best copy available. Nevertheless, items of marginal re prod udb0i|y are often encountered and this affects lh6 
0 of the microfiche and hardcopy reproductions ERIC makes available na the ERIC Document, Reproduction Service v^DRSji 
^[^(]]s not responsible for the quality of the original document. Reproductions fuppbed by EDRS are the best that can be made from 



AN EVALUATION OF NON-FORMAL 
EDUCATION IN ECUADOR. 

VOLUME II: OVERVIEW AND EVALUATION PLAN 



Luis M. Laosa 
Marvin C. Alkin 
Peter White 



U$ DEPARTMENT OF HEALTH 
EDUCATION tW^LFARE 
NATIONAL INSTITUTE OF 
EDUCATION 

'MS DOCUMENT WAS BEEN REPPO 
OWCEO EXACTlVAS PECEwEO PPOM 
TME PERSON OR OPtiANJZATiON OR»GiN. 
AT.NG'7 POiNTi O^^ V)EW OR OPiNtONS 
STATED 00 NOT NECE$SAR«LY REPJ?€- 
SENT Of P»C"AL NAT«ONAl institute OP 

eoyCATtON POSIT ion'or pol»cy 



The authors wish to acknowledge thfe ^collaboration and 
administrative^ coordination provided by Srta. Pilar Nunez, 
and the technical assistance ^provided ):)y^ Srt^. Piedad Figueroa 

and Sr. Anibal ^Villacis . ^ V 



Final Report AI0/t^-112/i 
July 31, 1975 . 



2 - 



ACKNOWLEDGEMENTS 

A project of the magnitude of the present one could not have been 
accomplished without the assistance and cooperation of a multiplicity of 
individuals and agencies. 

The authors are .g*>airef ul to the following individuals who served as 
evaluation field workers or animadores , throughout the field phase of the 
evaluation: {5* 
Nancy Llugui'n Julia' Barba 

. • Jorge Proano Marcel o Unda 

^ Kleber Ruiz Jorge Frei^-e 

Other individuals served as animadores for limited periods of time. 

We wish to thank Myron Vent and Bernie Wilder of AID Washington and 
Jon Gant and Jim Frits of AID Mission Ecuador. We are grateful to the 
Ecuadorian Ministry of Education and to the University of' Massachusetts 
Non-Formal Education Project personnel for their cooperation in the conduct 
• of this evaluation. • Special thanks are due Patricio Barriga for sharing 
with the project staff his valuable insights regarding the non-formal edu- ' 
cation project. We are grateful, too» to James Hoxeng for consultative 
' assistance during the planning stages of the evaluation design. W^tsh^to 
thank Jack Krakower for his expedient and able assistance in data analyses. 

We are thankful to Patricia Goetschel , Beatrice Kass, and Denise Minden. 
project secretaries. 

And mucjias gracias, to the hundreds of rural villagers and other indi- 
^ viduals throughout Ecuador whose interest, cooperation^nd' hospitality made 
this evaluation study the profoundly human experience it has been. 

. , ■ -MCA 
O • - ^ 3 LML ■ ^ 

ERIC . ^ pw 



TABLE OF CONTENTS FOR 
THE PRESENT VOLUME 



Chapter I - Overview 

Introduction I-l 

Objectives of the Evaluation 1-3 

Chapter II - The Evaluation Plan 

-The UCLA Intervention Study IN2 

UMass Impact Study IN5 

Materials Selected for UCLA Field Imple- ' II-8 
mentation and Evaluation 

Instruments 11-15 

Staffing 11-45 

Implementation Procedures 11-55 

Field Supervision and Quality Qpntrol 11-69 



References 



4-, 



CHAPTER I 

4 



OVERVIEW 
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INTRODUCTION 

This document represents the final report of the evaluation of the . 
Ecuador Non-Formal Education Project, this evaluation was conducted under 
Contract No. AID/ta - C-1124 with the U. S;. Agency for International Devel- 
opment. 

Tlje Ecuador Non-Formal Education Project began in the spring of 1971 
when a team from the University of Massachusetts (UMass) visited and stud- 
ied nearly thirty different programs in Ecuador which were engaged in non- 
school educational activities. The goals of the study were to identify on- 
going projects, to assess the potential of existing institutions 'to make 
use of non-school educational techniques, and to assess the willingness of 
these institutions to try out new procedures. The study confirmed that 
there were a substantial number of institutions either involved in non- 
school educational activities or interested in becoming involved. 

In the summer of 1971, a contract was signed between the U. S. Agency 
for International Development, the Center for International Education at 
the University of Massachusetts, and the Government of Ecuador. Emphasis 
during the first year of the contract, which becan in January 1972, was 
on the development of new techniques. During subsequent periods the em- 
phasis was to shift to discovering ways for efficient distribution and use 
of the new approaches (Evans and Hoxeng, 1973). 

, The individuals and institutions undertaking the Ecuador Noit-Formal 
Education Project shared a common ct^n'cern that a critical lack of human 
and economic resources existed to bring educational facilities to rural 



areas where some, l<ind of educational enhanc^ent appeared needed. They 
saw that educational facilities ip Ecuador were inadequate in reaching 
the full populace and that the materials utilized lacked relevance to rural 
people (Swanson, 1973). 

In the summer of 1974, 'the present evaluation study reported herein, 
was commissioned by the United States State Department's Agency for Inter- 
national Development. 
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OBJECTIVES OF THE EVALUATION • 

As stipulated in the Statement of Work of the Genefal " Provisions of 
the Contra(^ti the objective of this contract was to ev,aluate the non-formal 
ecTucatixin project in Ecuador and ta determine its replicability in other 
regions oV the world. The primary focus of this evaluation is "on the fac- 
tors in the instructional materials that seem reasonably related to intended 
and desired consequences in participants as individuals and socia*! groups." 

The general questions that this evaluation attempts to answer, in re- 
sponse to contract stipulations, are as follows: 

1. What kinds of effects (changes) can be observed; and 

2. What are the characteristics of the materials and procedures as 

^^'^-^ ^hey can be logically and/or empirically related to the desirable outcomes. 

K 

V 

A principal step in the development of the evaluation design was to 
articulate the general questions stipulated in the'contnact into more speci- 
fic questians in order to assure that the data collected through the evalu- 
ation design would provide information optimally useful for decision-makers. 

After a series of interviews and written communications with AID offi- 

cials in both the Washington Office and the Mission Office in QuTto and 

with University of Massachusetts Non-Formal Education Project personnel, 

» 

both in Amherst and in Quito, the following major questions wefe chosen as 

sir; 

those whose answer would provide the information needed by decision-makers 
at the various levels (U. S. AID Washington, U. S. AID Mission to Ecuador, 
Ecuadorian Ministry of Education): 
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1. What does the University of Massachusetts non-formal education 
purport to do? 

2. How dpes the UMass non-formal eclucation project go about imple- 
menting its goals and objectives? ^ . 

3. To what extent is the UMass non-formal ^ucation project carry- 
ing out its i/tended objectives effectively? 

4. What .are the learning outcomes of selected educational materials 
developed by the UMass non-formal education project? 

5. What* are the ^characteristics of the more and the least effective 
materials developed by the UMass non-formal education project?/ 

6. What are the characteristics of materials that wor^ well with 
learners of what characteristics? What> are the 'best matcKings? 

7. What facilitator/teachgr variables affect the.felative effective- 
ness of various materials? ' - ^ / . 

8. What are the njotivational attributes of^each of the educational 
games selected for in-depth Experimental analysis in the evaluation? 

9. What changes in attitudes and behaviors are produced by each of 
the educatiorfel games on a short-term basis? 

, lOl. What sequencingi, factors or prerequisites are important for the 
non-formal education games? 

11. What are the effects of replay frequency for each of the selected 
non-formal education games? 

12. What is needed to develop effective non-formal education* material 
and programs in countries similar to Ecuador? 

13. Can the project be replicated in other countries? If so, what 
conditions are most necessary to ensure success? 
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The evaluation design was devised in such a fashion as to obtain infor- 
mation that -would answer as objectively as possible the 13 questions pre- 
viousTy specifipH. As such, the evaluation design has two components. 

1. The experimental field implementation and' concurrent evaluation of 
the UMass non-formal education games that are among the most widely used or 
most widely accepted education games (the UCLA intervention study). 

2. The UMass non-formal education intervention impact study. 

The general designs of the two major components the evaluation are 
described in the following sections. 




The shaded areas in the map of Ecuador are the 'three 
provinces Tungurahua, Chimborazo, and Guayas — in wliich 
the 31 rural communities in the evaluation sample were lo~ 
cated. As can be seen, a very large geographical area was 
encompassed by the evaluation described herein. 
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-^^HE UCLA INTERVENTION STUDY 



This portion of the evaluation design was developed and implemented in 
order to prq^ide information toward answering the evaluatiirn questi^^s 4 
through 13 described in the previous chapter. 

In order to provide accurate data concerning the above evaluation ques- 
tions, it was necessary to observe and measure in a controlled manner, from 
its inception, the introduction and implementation of selected educational 
materials following various sequences of introduction, in populations of 
various characteristics, and by leaders of various ^characteristics'. Given 
the fact that the evaluation was not requested until after the UMass project 
had^assed its development, introduction, and implementation phases, and 
sfhce several of the evaluation questions call for planned variations in in- 
troduction and implementation of selected materials, it was necessary to pro- 
vide in the evaluation design an experimental field implementation of the 
selected materials to be, carried out by the evaluators considering the condi- 
tions previously mentioned.- 

Concomitant with this planned introduction and implementation of the 
selected non-formal education games, the objectives of each game, as"well as 
possible unintended effects, were measured in a pre-post fashion. The vari- 
ations in the introduction and implementation of the games, the character- 
istics of the leaders introducing the games, and the characteristics of the 
individual participants and of the respective communities, were carefully 
and systematically documented and measured. 

The procedures followed for the experimental field phase of the -Valu- 
ation for each community may be depicted in a flow chart as follows. The 
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flow chart is to be read- from left^to right, indicating the chronological 
sequence of events. 



Community 
Selection 


> 


Obtain 
Communi ty 
Acceptance 


> 


Sel ect 
Subjects 








> 



Play 1st 
Game in the 
Sequence 



Play 1st 
Game in the 
Sequence 



\ 




Individually 
Posttest 
(Second Test) 



Play 1st 
Game in the 
Sequence 



P/fay 2nd 
Sequence 



> 


Individually 
Pretest 


> 


Fill out 
Cofmuni ty 
Demographic 
Profile 




Play 1st 
Game in the 
Secii^uence 








> 


> 



> 


Play 2nd 
Game in the 
Sequence 

'2 




Play 2nd 
Game in the 
Sequence 




Play 2nd 
Game in the 
Sequence 

^4 


1 




— > 


— > 


> 



ERIC 



> 


Play 2nd 
Game in the 
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Demographic ' 
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The examination of the sequences depicted in the preceding flow chart 
Indicates a number of activities, each of which is more fully described in 
later sections of this chapter, but for overview purposes we will discuss 
the flow chart briefly. The first activity engaged in was the community 
selection. Recognizing differences between regions, communities were se- 
lected in both the Coast and Sierra regions and in a manner that provided 
us with communities similar to those utilized in the original tfMass inter- 
ventions. 

After communities had been 'selected, field workers visited the communi- 
ties^ito obtain acceptancer, a process involving a considerable amount of ef- 
fort. Individuals were selected to be participants in the intervention 
study. Simultaneously, other communities were selected as control communi- 
V^ties v/ith some of their residents individually tested. In both cases, par- 
tici pants vfere given ind^idual pretests consisting of demographic data, 
literacy 5kills, numeracy skills, and critical consciousness. In addition, 
evaluation field workers completed an extensive community demographic pro- 
file. Following the completion of all of these "pretest" instruments, the 
first game introduced into the coirenunity was played in sequence five times 
with intervals of at least one week between playings, A second round of 
individual testing, including all of the individual tests previously des- 
cribed, was administered after the first game sequence, A second game, 
played in' sequence five times was also followed by individual testing. Also 
at .the end of the total intervention, a community demograph]^c profile was 
again completed. 




1 1-4 



UMASS IMPACT STUDY ' 

The portion of the evaluation designed to assess'tTte relative impact 
of the UMass project in eight selected rural cqmmunities had three 'compo- 
nents: 

V 

if a detailed "impact profile" for each community, which would gather 
quantitative information on the processes and materials used by 
the UMass project in their intervention in the communities. ' 

2) a qualitative analysis of UMass intervention, based on personal 
observations and detailed interviews with facilitators, game 
session participants and community residents. Here the focus 
was on the UMass project intervention as seen by the people in- 
volved, and its effect (if any.) on them personally and on' the ^ 
community as a whole. 

3) individual testing of game session participants. Between 6 and 
10 people in six of the eight communities studied were given the ' 

, same individual test measurement that was applied in the UCLA ex- 
perimental and control communities. In two of the^ communities 
;the gagies had been played so few times (between 3 and 4) and with 
'SO few. people that no individuals could be found of whom it could 
be said that there had'been a personal impact. 
Our objective with this study was to identify and analyze those pro- 
cesses— introduction to a community, selection and .training of group dis- 
cussion leaders, participant characteristics, follow-up and support—on 
which much of the success of a non-formal educational project depends. 
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We selected ';pight conmunities» four in the highlands -and four" on the 
coast in which the UMass project personnel and trained facilitators had used 
at least some of the educational games. The four highland communities - 
Punachisac, Tutu'pala, San Martin, Baldalupaxf - were .selected on the basis 
of their "success" as determined by UMass non- formal education (NFE) pro- 
ject documents and reports, and by conversations with UMass project person- 
nel* The four coastal conmunities we studied, were also selected on the 
ir "success." <a — 

The "impact profile" allowed us to measure and then compare with the 
other villages in the study the degree and 'intensity of UMass NFE interven; 
tion imearh. This informatioji was used -as a backdrop to the in-depth ob-. 
servational and interview data; togethe'r they would answeRo^valuatfon ques- 
tion #3 concerning the effectiveness of the implementation of UMass project 
objectives. - - '.r - 

People In six of the communities who had participated in the game ses- 
sions there were then given the Individual pretest applied Jn the UCLA ex- 
perimental and "control" communities. Their scores would be compared with 
the "control" communities in a *'post" comparison on the assumption that' 
they were a comparable selection and mix of people as in, other rural vil- 
lacfes in Ecuador represented by the control commun/Tfies. In addition, they 
were each Interviewed about their personal perceptions of the non-formal 
materials and their utility in daily life. 

The impact study attempts to analyze the UMass project as a multi -level 
interaction process: at one' level are the concepts, plans and documents 
that spell out intent and procedures. The next level is that of immediate 
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Conmunities Selected for the UMass 
' Impact Study and Number of Participants 
Administered the Individual Tests 

Number of Par- 
Conmunity ' ticipants lested 



Sierra 



PunachTsac 



8 

Tutup^la V ® 



'San Martin 




/ 8 

Baldalupaxf - ^ 



Coast 



Colonche ® 

•San. Pedro - ® 

'sinchal ^ 

Valdivia ^ 
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project impact: implemenfition of the plans and goals through the training 
of the community facilitators in the use of project materials and project 
ideas. Finally, the project as it is interpreted by the facilitators which 
is then passed on to individuals in each of the rural communities who, in 
turn, process 'that information and reach their own highly personal conclusions 
The impact we study in this particular component of the evaluation is not 
, merely that of certain materials on certain kinds of people, but rather of 
the whole series of interacting ideas, personalities and processes that 
determine the effectiveness of a non-formal education project in rural areas. 

) 
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MATERIALS SELECTED FOR UCLA FIELD IMPLEMENTATION AND EVALUATION 



The evaluation questions'which address the component of the evaluation 
design described in this section of the report refer to "educational materi- 
als selected for in-depth experimental analysis." As h.as already been 
, pointed out, in order to answer the evaluation questions, the evaluators 
had to actually, introduce and implement-in 'addition to measure the effects 
and document the process-selected non-formal educational materials as treat- 
ments. This Was necessary in order to evaluate the materials in the manner ' 
suggested by the evaluation questions. The following non-formal education 
games were selected for planned sequential field implementation. and concomi- 
tant evaluation: 

a) " Hacienda (Game of Life), Sierra -Version 

b) Hacienda (Game of Life), Coast Version 

c) Number Bingo (addition and multiplication) 

d) Syllable Dice 

e) Syllable Cards 

The criteria f^r selecting these particular games for field implement, " 
tation and evaluation, were: (a) that the games be among the most popular 
and widely used of all the material s> developed by the UMass project; (b) 
that as a group they be representative of the major areas of learning on 
Which the UMass project focused (Critical Consciousness, Literacy, and 
Numerical- Skills). ' . / , 
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Description , ✓ 

* 

A description of each of the non-formal education mai:erials selected 
for the field implementation and evaluation is presented below. 

V 

Hacienda (Juego de la Vida) . ^ ' 

"Hacienda," often called "The Game of Life," is a simulation board 
game based on the popular Monopoly. But rather than streets, hotels, and 
* utilities. Hacienda revolves around rural South American reality; peopte 
buy and sell the typical agricultural products (those of the highlands in 
the original version, and tropical products in the coastal version), deal 
with local authorities (priest, political chief, lawyer) and experience 
the frustrations of a peasant's daily life. 

The object of the game is to get the peasant players to improve life 
by taking advantage of the opportunities available: information, organiza-' 
tion, cooperation. Designed to simulate real life in order for t-he campesino 
to "back off" a moment to. see his own situation objectively, a key component 
of the game is the discussion which follows play; the game activity and the 
reflection on it are designed to lead to a state of "critical* consciousness" 
( "concientizacio^" in Spanish). 

"Cri^al cons(ffousness" is a concept based largely on the writings" by 
and about Paulo Fneire and the literacy method he developed in the late 
1960"s. It refers. to the ability to objectively describe one's own situ- 
ation,. to analyze it in terms of one's goals, and to plan concrete steps to 
. reach those goals. It represents an initial step toward overcomin^J^^res- 
sion." It deals wit'h the relationship between one's perception of a situ- 
ation and the objective situatiQi^^^itself. . 



t 
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Hacienda is described in detail in Hacienda Technical Note #3 by J. 
Hoxeng, 1973. A synthesis of Freire's theory and methodology can be found 
in Concientizacao and Simulation/Games (Smith, 1973). ~ 

Syllable Dice 

This set of between eleven and fifteen small wooden blocks is what 
the UMass project calls a ''fluency" game—one designed to develop or in- 
crease a specific literacy or numeracy skill in the individual who uses 'it. 
Each block has a single letter or a syllable on each side (the original 
version used only letters; field experience and suggestions from the Di- 
rector of the Ministry of E,ducation's Adult Education division led to the 
use of syllables since Spanish is a highly regular, syllabic language). 

The game is easy to play', and lends itself to any number of variations. 
Playing individually or in groups, the dice are tossed and words are then 
, formed combining those syllables and letters which fall face up. The spe- 
cific skill which it aims to develop is the correct- identification of let- 
ters and their formation into correctly spelled words. It does not lend 
itself as easily as Hacienda to the process "Reading to "critical conscious- 
. ness," but does allow for discussions of daily life situations based on the 
words that are formed. To the extent possible, t"he game (the selection 
and various combination of syllables) was designed tq turn up words that 
are common in rural Ecuador. ' , 

A full explanation of the development and design of this game can be' 
found in Letter Dice Technical Note #6 by J. Hoxeng and A. Borja, 1973. 

. 23 ' • ' 
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Syllable Cards 

Another "fluency" game, Syllable Cards closely resembles its prede- 
cessor. Syllable Dice. It is a deck of some 80 cards, roughly the same 
size' and consistency of regular playing cards; each has a syllable or single 
letter profited on one side. / 

Its aim is to increase the players' abiUlp to quickly identify letters 
and sounds, and to form them into words.' Dealt out to the participants, 
jJlShr^ies to form as many different words as possible. The cards are ver- 
satile, and allow the pla^i^r^to invent a number of variations on this way 
of playing; "rummy" is a n^me attached for convenience, since it is a common 
game in rural Ecuador and in no way limits the cards' use. (In Spanish, the 
game is usually referred to a " Naipes de' Sflabas ." or "Syllable Cards. "^ 

The game also has a certain poten^^^as a discussion tool; the words 
that are formed (the "syllables having been chosen 'by the UMass project to 
facilitate the formation of words common to rural Ecuadorians) are focused 
upon as representatives of a larger reality, and these ideas ^re then dis- 
cussed as they touch upon daily life. 

Number Bingo % ' 

Number Bingo is a "fluency" game designed to increase functional arith- 
metical skills..^ Closely resembling the familiar bingo used in the United 
States, it is a series of cardboard playing boards, each divided into squares, 
and each square containing a number. A "reader" calls out math problems to 
the participating group, and the player then tr>es to locate the answer to 
^he problem on his board. There are both addition and multiplication ver- 



sions of the game, and both were used by the evaluat'^n." * 

The Bingo game is designed to be used with lea5*hers of different levels 
of knowledge. 'The cards containing the .computational problems to be read 
out also have the answer printed on the reverse; with people unfamiliar 
with numbers, the computational proJ?lems can be called out or shown, the 
^ participant having only to/ identify the numbers on his or her board. For 
the more advanced student, speed becomes the object since this is an impor- 
tant skill in the rural markejfc places. 

As with the other "fluency" games, UMass project bingo can be used 
as a discussion-starter, beginning wjth the utility of numerical sldj-ls 
in rural life and moving on to one's situation vis-a-vis the market. 

A detailed discussion of this^gam^can be found (in Number Bingo: Tech- 
nical Note #7 (Gunther, 1973), \ 

• . , ■ > 

Game Sequences 

^ In some of the communities we selected for our implementation and ,evalu- 
ation of the non-fcrfmal education games, w.e introduced the games, two per 
^ community. In other of the communities in the sample, w6 introduced and 
evaluated only one game. In instances where two games were introduced' the 
following sequences were^used:^ 
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Sequence Code' Game Sequence ^ > ^ 

A. Hacienda (Game of Life) - Number Bingo 

B. Hacienda (Game of Life) - Syllable Cards 

C. Hacienda (Game of Life) - Syllable Dice 
d/ • Syllable Cards - .Stable Dice 

The^ number of games in a sequence and the number of two-games combinations 
included in this portion of the design had to be limited to the above four 
different sequences, given the temporal and financial constraints inherent 
in the present contract. These four xdiffemii^quences, however, are con- 
, sidered to be adequate for providing information re\vant to the evaluation 
questions. 

Table 1 presents the communities selected by the present evaluators for 
introducing, -implementing, and ^^uating the non-formal education materials, 
and the material or materials sequence employed in each community. Communities 
were assigned randomly *to games and game sequences. ^ . ^ ^ 

As shown in Table 1, there were total of 23 rural communities selected 
by^the eva/uati on staff . Of these 16 were in the Sierra , or mountain region, 
. and 7 in the Costa , or coastal region, of Ecuador. 

It should be noted that Ecuador is geographically and culturally divided 
. into three distinct regions: the Sierra , the Costa , and the Oriente , or 
Amazon jungle region. ^The UMass non-formal education jiroject has operated 
in "^0 of these r^ions: Sierra and Costa. The people from these two 
areas vary widely with regard to their culture, life style, dialect spoken, etc. 
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In 11 of the 23 communities, two non-formal education matenals''^vere 
Introduced and completed (in, 5 different sequences); in 11 communities 
a single material was introduced. In the latter group, the game sequence 
vras completed in 4 of the 11 communities. In one community, Jime'rlez, the 

* evaluation project completed -only the "pretest" instruments; the partici- 
pant group then decided the community didn't need an adult^ucation pro- 

, gram after all. 





Table UA 

Conmunities Selected by the Evaluators for Introducing, Implementing, 
and Eva-luating the Non-Formal Education Materials, and the Material 
or Materials Sequence Employed in Each Community 



Community 

Sierra 

' Rumipamba 
Hipolon^o^ 
Yayuligiii 
San Antonio 
San Andre's , 
San Pedro ' j 
San Franciscp 
Urbina 
El Calvario 
Asuncion 
La Libertad 
San Isidro 
Pilchipamba 
Mi raf lores 



Number Bingo Hacienda Syllable Dice Syllable Cards 



2 
1 



2 
1 



Chil^co La Esperanza 
Jimenez (no number 



1 
1 

2 
1 



2 
2 



1 
1 



1 
2 



r) 



1 
2 
1 



Coast * 

01 on 
Palmar 
San Pablo 
Barcelona 
Montanita- 
Dos Mangas 
Cadeate 
J 



1 
1 



2 
1 



Note: 



A "1^' and "2" indicate whether the non-formal education materials was first 
or second in the sequence. 
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Table IL2 



Number of Persons Administered the Pretest in Each 
UCLA Experimental Community^ 



Community 
Tungurahua Province 



Pedr^ 
Andres 



San 
San 
Urbina 
Rumi pamba 
Hipolongo^ 
Yayuligui 
San Antonio 
San Francisco 
El Calvario 
Chi 1 CO La Esperanza 



Chimborazo Prpvince 

Asuncion 
La Libertad 
San Isidro 
Mir^flores 
Jimenez 
Pi Ichi pamba 

Guayas Province 

Olo^n 

Palmar 

San Pablo 

Montanita 

Cadeate 

Barcelona 



Number of 
Partici 



13 

15 

17 

20 

20 

23 

16 • 

17 

25 

21 



18^ 

15 

16 

18 

10* 

22 V 



15 
17 
17 
15 
15 
27 



Total 427 partTci pants 



* intervention discontinued 
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INSTRUMENTS 

There were two basic instrumisnts used in the performance of this 
cdmponent of the evaluation. The first is a cornmunity demographic profile. 
This instrument, admitiistered in each community in which the non-formal 
education games were introduced by the evaluation staff, obtains detailed 
data on the demographic, social, political, economic, racial, linguistic, 
religious, an'd other relevant characteristics of the community. , 

The second is a set of individually administered tests given in a pre- 
second-post basis to each subject participating in the experimental portion 
of the evaluation. The intervening treatment was the non-formal education 
'game led by the evaluation field worker five times in that community. The 
second test administration occurred following the use (5 times) of the. first 
game and the third test was administered after the second game was played 



5 times. These individually administered tests cover the following area< ^ 
which are directly related to the objectives of the five non-formal educa- 
tion games. 

1. Literacy 
^. Numerical Skills 
.3. Critical Consciousness (Attitudes) 
In addition, data were collected from each participant to obtain in- 
dividual characteristics. This individual demographic questionnaire was 
administered on a pre- and post-basis. Furthermore, extensive work diaries 
were ^maintained by all field staff to provide in-depth and systematic quali- 
tative data on ^1 field implementations. Additional instruments were de- 
vised for use in'the UMass impact study. 
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The Community Denwqraphic Profile 

This instrument was designed to identify and' measure numerous factors 
—ethnicity, economic base, internal organization, external contacts and 
influence— that would have a bearing on the development of an extracurric-. 
ular adult education program ia selected rufal communities. It was applied 
once in each village in which UCLA attempted to introduce the non- formal 
education games,' and again a second time in those communities where a 
second game was played five times. The same instrument was applied onte in 
eight selected communities identi^J^ed as ."impacted" by the University of 
Massachusetts project. 

The profile is six pages long, with clos6 to s.ixty iten^^ee Appendix 
A).. The information it seeks is straightforward, and can be obtained by in- 
dividual observation on the part of the field worker and interviews with 
local authorities. It was designed with the assistance of Ecuadorian tech- 
nical consultants, then reviewed item by item with the field workers to 
identify possible areas of misunderstanding or misinterpretation. ^^&n|JlTasis 
on the training of the field workers was placed on the need for accurate, 
reliable information; certain quantitative data (e.g., number of radios in 
the community, population, number of illiterates) would have to be based on 
interviews with various contacts in the community and, where possible, veri- 
fied with available official datcy-iuch as the census. 

The experience d?^ field testing the instruments revealed a variety 
of interpretations of certain items and the neeessi.ty of constantly asses- 
sing and considering the reliability of each data source utilized. An ex- 
amination of several- selected items will provide a fuljer understanding of 
procedures. ^ 

31 

11-16 ■ 



Item 6 . Ethnicity of community. - • ^ 

This was tc^be judged by the field worker, based on forms of dress, 
and local customs.' The entire coitounity was to be considered, and not 
just that segment participating in the' UCLA program. Questions on this 
point could be directed to local leaders, but their persona] bias could limit 
the reliability of their answers; they were used more commonly as "checks" 
•on the worker's obse*ii/a^:ions. 
Ktems 8 and 9 . Number of inhabitants and houses. 

To be determined by interviews with local authorities— teniente polftico 
priest, teacher, community leaders, with a reasonable approximation to be 
. made where differences appeared* In some cases exact data from the recently 
completed nationa-l census were used,* There was no confusion as to the popu-"^ 
lation of this or that community, nor as to what constituted a house. 
Items 10 and 11 , People over six years of age who can read and write; 
number of illiterates. 

Determined ^y interviews with local leaders, especially the school- 
teacher, after careful explanation of what was considered to be .literacy: 
ability to read and wr.ite one *s name and perhaps simple sentences. Responses 
varied widely, from "I have no idea" to "everyone can read and write," Re- 
sonable estimations were made based an the interviews and personal obser- 
vations. 

It em 12 , Services in the community, 

. ^d, number of students enrolled in the school (s). 

Determined by interviews with the teacher or the president of the 

Padres de Familia , Sometimes neither were sure, and a close esti- 
— ■ " ^* 

mation would be accepted. 
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f. car-worthy" roads 

Roads on which cars could and did travel > used for comnuni eating 
with other communities or towns, 
g* non-carworthy roads. 

Exceedingly difficult to determine with exactitude because , of the 




multiplicity of narrow trails, small roads, and footpaths tha^ 
criss-cross a community and sometimes a whole area. Included in 
the final figure were thQ$e that were used to communicate internally 
and externally between groups of houses or areas or neighboring com- 
munities. Paths leading to individual's houses were discounted, 
k. public health. 

^t have to be a physical location; weekly or even monthly visits 
by a doctor or nurse were counted as a single service, as long as 
it was on a regular basis. 
Item 13. Private businesses. ^ 

Determined b/ interviews and observation'. They could be individually- 
housed businesses (such as a store or bar), form part of a home,. or merely 
be an occasional household activity (such as shoemaking .or carpentry) as 
long as it occupied an identifiable area within the house and served the 
community in some way. 
Item 14r Projects underway. 

These had to be projects that had taken on an organizational or physi- 
cal form (i.e., constituted committee, foundations laid, money raised, etc J 
and were in operatioli at the time of UCLA intervention. Plans, ideas,^ and 
projections for- the future were noted but not counted. 
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Itemje. Relations between religions. 

Determined by observations and informal conversations. In most cases 
the community would be homogerieous-entirely Catholic--and the relations 
(in reality non-exTstent) would be indicated as harmonious.' 
Item_17. ^ Organizations within the coimiunity. 

Those that were in existence and functioning in some manner at the 
time of UCLA intervention. An organized unit would be counted-sports clubs 
for example-but not groups of people that get together irregularly for 
fleeting purposes. 

ItenUS. External organizations in the conmunity. 

Counted were all the organizations (except UCLA) external to the com- 
munity that were currently active in some way theref their presence did not 
have to be permanent (such as the Ministry of Education) but it did have to 
be regular (e.g.. periodic visits by a medical" team, sponsored by the Ministry 
of Agriculture), 

^tem 21 . Language used in corrmunity. ' , ' 

Determined by personal -observation and informal conversations. Direct 
qtiestions on the matter could draw evasive or false answers. aTid" were only ^ 
used as a "check" on the observations of the field worker. 
Item_22. Number of radios in the cormunity. 

Determined by numerous individual interviews. No one coutd know for 
sure, nor was a "census" possible; aVeasonable extrapolation from all re- 
sponses was accepted. - 

Item_24. Years since the formation of the community. 

. In very few cases was it possib^ to determine this with any accuracy. 

s 
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The largest figure obtained, gleaned from "oldest living memory" or "my 
grandfather ha_d said..." would be accepted. The question refers to the 
formation of the community as an identifiable entity and not to its legal, 
chartered formation. 

^^^"^ (page 5). Types of newspapers available in the comnjunity. 

This was interpreted as "number of different newspapers" rather than 
"type," since standard popular newspapers are the only kind available. To 
be counted, they would have to be sold in the community, even if only once 
a week, and not merely brought in by people who had bought them in neigh- 



Donng towns or cities. 



Item 25 . Types of magazines read in the community. 

Here there was not the need for them to be directly sold in the community, 
rather that they merely be available (i.e., brought in from outside, passed 
on from person to person). The emphasis was on type of magazine— news and 
political would begone, general interest and features another, "fotonovelas" 
a third, and comics yet another. 

Item_26. Types of transportation used to leave the community; 

Included were those means regularly available (even, if only onnea 
week) or commonly used, with the emphasis on "to leave the community," and • 
not just for Internal visiting or intracommunity transportation of products. 
Page 6 . 

a. Number of times UMass games used with animador's (UCLA evaluation 
Ecuadorian field worker) presence.' ; 

Included are the five formal sessions with each game; noted 
elsewhere— in the diaries and sometimes, as an aside on this same 
page— are the rare occasions when the animador was present, but not 
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in a directive role, when the game was played informally in the 
coiMunity. 

b. Number of times UMass games used without animador's .presence. 

In most cases it was easy to determine exactly how many in- 
dividual times the games had been lised among the participants alone. 
I In others, they had been tal^en home and use.d several times with 

different groups; in these communities, we took as our figure the 

highest number of times the games were played by any one individual 

u ' 
outside the regular UCLA sessions. 

Items a_ and b inmediately above 'were completed at the end of each 
UCLA experimental intervention." 



Criterion Referenced Testing 

An important pofnt.must be made at this juncture about the nature of 
the tests constructed for measuring literacy and numerical skills.^ Thes? 
tests were constructed as "criterion referenced" tests. Alkin (1974) has 
identified the following three definitions of "criterion-referenced tests" 
in the literature: ^ ' 

(1) ""A criterion-referenced test is one that is deliberately constructed 
to yield- measurements that are directly interpretable in terms of 

specified performance standards Performance standards are 

generally specified by defining a class or domain of tasks that 
should be performed by the individual" (Glaser & Nitko, 1971, 
p. 653). 

(2) "A pure criterion-referenced test is one consisting pf a sample 
of production tasks drawn from a well-defined population of per- 
formances, a sample that may be used to estimat-e'the proportion 

• of performances in that population at which the student can suc- 
ceed" (Harris & Stewart, 1971). 

(3) "Criterion-referenced measures- are those which are used to as- 
certain an individual's status with respect to some criterion. 
I.e., a performance standard" (Pophafn & Husek, 1969, -p.-^). 
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While these definitions differ considerably in terms of the limi- 
tations and constraints placed on a criterion-referenced measure, they 
'all share a common emphasis on two characteristics. First, each defini- 
tion emphasizes test organization (i.e., test item selection) based on- 
specific tasks or behavioral objectives. Second ,^_eadl definition empha- 
sizes assessment in terms of predefined performance criteria. What is* • 
important to note about this discussion is that the project "team was 
concerned with' developing tests that determined the extent to which par- 
ticipants met an "absolute" standard (fulfilled the objectives) rather 
than a test that utilized a "relative" standard. Tha> is, the critical ' 
question in our study is whether community participants attained the 
objectives specified as goals'-in the game materials. 

Literacy and Numeracy Skills 

The major objective of UMass non-formal "fluency" games is the trans- 
fer of literacy and numeracy skills to their users. Hacienda, though a 
"simulation" game with no stated cognitive objectives, includes many ele- 
ments that involve the use of literacy and numeracy ski.lls already possessed 
by the rural villagers. In order to measure the possible gain in these 
skills through game use, we identified the specific skill areas at which 
the four evaluated games aim and designed a criterion-referenced, .test based 
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on those skills. The objective of the ^'fluency" games are to develop 
first-grade to third-grade level operations of word formation and math 
problems (as reflected in the curriculum of the Ecuadorian primary school 
system). The objectives of the Hacienda game are somewhat higher (third- 
grade to fifth-grade level), related to the reading of simple sentences 
and more complex math problems, both^al and written. ' 

The measurement of possible gai^in literacy was divided into three 
sections: 1) identification of individual letters, 2) identification of 
individual syllables, and 3) identification of individual words and words 
formed into sentences.. 

The measurement of possible gains in math skills was divided into 
three s^tions: 1) identification of individual one-, two-, and three- 
digit numbers; 2) oral math operations (addition, subtraction, and multi- 
plication); and 3) written math problems (covering addition, subtraction, 
multiplication, and division). > . 

No one of the four evaluated games, nor any one of five game 'sequence 
combin'ations employed, touches upon all of the skill objectives. The same 
test was applied to all the evaluation participants, however, in order to 
be able to measure skill gains related to the specific game(s) being 
played in that 'community as well as measuring unexpected skill gains 
brought ^bout by the various games. 

• Following the application of the tests in a few communities in the 
UCLA experimental sample, it was discovel^that the non-formal education 
sessions had a strong attraction for individuals with several years of 
-formal education. These individuals reached near-criterion level scores 
on the Literacy and Numerical Skills tests. Therefor?e, several items of 
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even higher difficulty but conforming to the objectives implicit in the 
materials were added to the Literacy and Numerical SkilH tests, raising 
the possible criterion attainment level. In subsequent ^communities, this 
version with the higher ceiling in these two tests was used- 
Six communities were administered Version 1 of the Literacy and Numer- " 
ical Skills test, and nine communities were administered Version 2. For 
purposes of allowing comparison on test scores among communities receiving 
different versions of these two tests. Version 2 tests were scored in two 
different ways: (a) by counting only those items which were part of its 
Version 1 counterpart and (b) by counting all of the items in Version 2^ 
It is helpful here to^differentiate between a test version- -l oV 2-- 
and a' test scored according to Version 1 or Version 2. scoring system. Thus, 
on each test or subtest involving Literacy or Numerical Skills, individuals 

s 

from communities receiving Version 2 of these tests have two separate scores: 
a Version 1 score and a Version 2 score--Version 2 scores having a higher 
criterion attaim^ent level than its V?j::sipn 1 counterpart. Of c'burse, in- 
dividuals from communities receiving Version 1, of the tests could only have 
Version 1 scores. ^ 

For the purpose of analyzing the data in order to answer the evaluation 
questions, the following scores were derived from each test: 
1. Total Reading. This is a composite of four reading subscores: a) iden- 
tification of individual letters out of fields of eight letters, b) 
identification of individual syllables out of fields of eight syllables, 
.c) identification of individual words out of fields of four words per 
section, ^d) reading sentences of various lengths, with each'word cor- 
rectly read counting as a point. 
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In Version 1 of the instrument, this section would have a possible 
total of 30 points; Version 2 had a possible 61_ points. 
2. Total Functional Writing. "Functional" is defined as readable or un- 
• derstandable; if a word could be read phonetically arid understood in 
Its context, it would be counted as correct regardless of errors in 
spelling, punctuation, or capitalization.- This section is the sum of 
three'' subscores: a) writing one's own name, first and last, b) writing 
individual words given the participant by the fieW worker (wherein 
each letter counts as a point), and c) writing sentenc^WiQtated slow- 
ly by the field worker (wherein each word counts as Jne poss-i¥^"p^t) . 
Version 1 of the test had a possible score of hj, V^r^on 2 had a 
- possible 44 points. 
3* Total Standard Writing* All the words written (except person's name) 
we^e then checked against standard Spanish; words had to be correctly 
spelled and capitalized, with proper accent marks, and clearly "^legible. 

A 

Points were given for letters in individual words and for each cor- 
rect word in the dictated' sentences.* 

Version 1 ^f^the-test had a possible 26_ points. Version 2 had 43^ 
possible points. ... 

4. Total Standard Literacy^. This is the cpmbinejl scores of Items 1 (To- 
tal Reading) and 3 (Total standard Writing) |bove, plus the person's 
name. It is a reflection of the participant's ability to read and 

\ write correct, standard Spanish* * ^ . ^ 

Version 1 had a total possible point score of 58, Versioir2-had 
106 possible points. 

5. Total Functional Literacy. A composite score of Items 1 (Total Reading) 
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2 (Total FunUixir^Jir^ above, this is a reflection of the 
individual's abiMty/^ -redid and write functional, legible Spanish 
without regard^^;^grammatical errors. 

Version 1 liad 57 possible points. Version 2 had a possible lg5 
points^ iv 



Total Math. Included in this composite s/CSrh ar4 the subsctJfeTTor 




a) identification of individual pne., two( and ^ree-digit numbers in 
several multi-numeral fields; b) addition problems read to the partici- 
pant and calculated mentally; c) subtraction p||blems read bynhe field 
worker and Calculated mentally; d) multiplication problems read to the 
individual and calculated m^tally; and e) math problems, ir/cluding all 
four basic functions, done by the participant in a written form working 
alone. ^ 

Version 1 had 32 possible points. Version 2- had 49 possible pt)ints. 
Critical Consciousness. " The twelve questions designed to measure the 
participant's perception of himself, others, and his community were 
given points according to the positive strength of each response. This 
score is the total point count for the' twelve questions. 

Versions 1 and 2 contained the same questions and were given points 
in the same manner. Each individual could scores^ maximum of 28 points- 
Draw-A-Man Harris-Goodenough Test. Designed to measure an individual's 
level of conceptual maturity through the use of a drawing ejcercise, the 
score is the sum of points given for identifiable features (eyes, nose, 
fingers, hair), details (hat, shoes), and other aspects of the drawing.' 

,^It was applied in the same Way in Versions 1 and 2. There are a 
possible 73 points total. 
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The tests used to measure the variables described in this section are 
presented in Appendix B'. 

Critical Consciousness 

One of the major aims of the UMass project was to raise, the level t)f 
"critical consciousness" of Ecuadorian peasants. The non.-formal education - ^ 
game of Hacienda has as its specific aim the development of "critical con- 
sciousness" among those who use it (cf. Hoxeng and Evans, not dated). What 
is "critical consciousness?" "Critical consciousness" is a concept based 
largely on the writing by and about the Brazilian educator Paulo Frefre 
(cf. Freire, 1970a; 1970b; Smith, not dated; Smith, Alschuler, Moreno, & 
Tasiguano, 1975). The concept of "critical consciousness" refers to the • 
ability to objectively descr-ibe one's own .situation, to analyze it in •- 

;jtenns pf one's goals, and to plan concrete steps to reach those goals. 
A group of researchers in Ecuador and at the University of Massachusetts 
(Smith, Alschuler, -Moreno & 'Tasiguano, 'l975) , have suirniarized and defined 
Freire's abstract concepts in order to allow investigators to translate 
cepts into operational definitions. They identified four "con- 
distinct but interrelated indices of critical consciousness" jj) 
the capacity to see one's situation as problematic and susceptible to change, 
in contt;asl to viewing one's plight as static, fated by God, and unchangeable, 
(2) the )^ awareness of social injusti-ce— how the situation is not as it 
should be, or where there are inherent contradictions, (3) the.,relationsh>p" 
to the oppressor— the rea'Uzation of \m oppres-sed people play hdst to the 
oppressor, collude in maintaining the status quo through inaction and^ belief 
in the inherent "rightness" of the oppressor, and (4) critical collaborative 
potion to transform the situations that put the dqpressed and the oppressor 
Into oppressive relationships. 
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Based on the above analysis of the concept of critical consciousness, 
a questionnaire was developed for purposes of measuring change in critical 
consciousness among,Jndi^jjdi^^ participating as subject in the evaluation, 
ThScfluestionnaire consists of a total of 12 items.. These are presented 
in Appendix B. 

Individual Demographic and Other Background Data , ^ 

^ The individual tests each ccfntain a section that measures a series of 

demographic and background factors, including years of schooling completed, 

attendance at adblt night school, newspapers read per week; access to maga- 

.1 

lines,' television ,^an?hnav^esv how often the radio is listened to; etc, 

The demographic data--sex, age*, occupation, marital state, ^number of 
children, ethnicity, religion, language—and the data on participation in 
community affairs were obtained in order to be able to compare the charac- 
teristics of our participant groups with the' characteristics of the comnun- 
ity at large. With this informatioA, we would be able to answer evaluation ' 
question #6 concerning learner characteristics vis-a-vis each of the mater^: 
ials. It would shed light as well as on the broader question of the-n^lative" 
acceptance of non-formal education programs in the rural areas of Ecuador: 
what kind of pe,cson is attracted to them? \ 

Application of the Individual Instruments 

The injjrument developed to measure literacy, numerical skills, and 
" critical ^consciousness, and to obtain demographic and background data on 
each individual was composed of three sections: 1) demographic and other 
background data (age, sex^ccupation, years of schooling, influence of 
outside media, participation in community affairs); Zl establishment of the indi 
viduaVs ability to perform letter and word identification, simple reading 
and writing, basic mathematical* operations— to wMch the UMass non-formal 
O nater.ials were directed; 3) elici t the resoondant's view of hirtiself, his 
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community, and the outside world, and his ability to affect or change his 

* 

environinent, through a series of questions on those subjects. 

It is a long questfonnaire-r27 pages in Version 2, with close to 100 
items and/or tasks for each participant. It measures each skill in a pre- 
cise manner. With the 'differences in each of the field ^workers (experience 
skill at asking questions, personal traits) and those that would be pre-' 
sented by each parti ciparit^in each community, plus the difference between 
communities and regions within Ecuador, it was of vital importance that the 
instrument be applied in the same way by all the workers and in all the com 
munities if we were to obtain relia^ble, general izable data. 

Tour days of the fifteen-day September training session were dedicated 
exclusively to a review and practice with the instrument. First, di^ the 
field workers understand each item? Would it be clear to a peasant? Sug- 
gestions were made and slight changes in wording were made. T^re followed 
art intensive application of the questionnaire between field workers, one 
acting as peasant and the other as interviewer. This was supervised by the 
coordinator and technical staff, with personal -differences in style and 
wording, and potential areas of misunderstanding or lack of clarity noted 
and later reviewed, item by item, with the group. The same process was fol 
lowed several times, until we were satisfied that the field staff was famil 
iar and comfortable with the instrument and could deal with problems that 
might come up in its application, ^ ; 

The next step was the field testing. Carried out in a village some , 
distande Outside Quito, the staff worked in teams of two to evaluate each 
other's performance in making the personal contacts and in the use of the 
quest^nnaire. As expected, the direct experience brought out a series of 
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minor problems-dtfficulties in certain items, unexpected answers by par- 
ticipants, leek of clarity with some of the wording, questions of style. 
0pon returning to the office, another .full day was spent reviewing the pro- 
cess once sgain,^ making slight alterations, and agreeing on a common appli- 
cation founded in reality as well as theory. 

An iteni'by-ltem review of the instrument and its application follows; 
Items 1-5, Identification of field worker, community and location. 
^ Item 6, Name of person.- The full name, with both last name^ in Case 
of confusion or father-son pairs of participants. ^ 
Item 7, Age, Stated injyears, to last birthday. If the participant 

was unjure, his or her estimate wot* Id .be accepted. 
Item 8, Sex of participant. 

Item 9. Occupation. A list of nine possible occupations is given, 
with th9t of the participant to be the one that is his or her 
' wajpy source of income . If he has a secondary occupation, it is 
to be written to the. side of the major occupation, but not marked 
9$ i response. 

Item 10. 'Ithnicity. This was a judgment of the field worker, based 
on physical features, customs in the home (if observable) and 
clothing Worn. 

Item n . Eduea^tion. Stated in grades of formal education completed . 
Incomplete grades, or years necessary to complete a certain grade 
were not to be taken tnto censi deration. 
Item U. Adult literacy classes. Any participation at any timg or" 

for any length of time would be considered an affirmative answer. 
•Item 13. Married state. The participant's response would be accepted 
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as stated, even when in variance with reality (married vs. free 

union, for example). 
Item 14.- Number of children. Surviving children, as infant mortality 

in the rural areas is high. 
Item 15. Persons living in participant's residence. The number of 
Vthe extended family and non-family, if that was also. the case. 
Iteip 16. Days per week the participant listens to the radio. The 
^ number of days in an average week that one listened to the radio, 

even casually; multiple listenings in the same day were counted 

as a single day. 

Item 17. Days per week one watches television. Total number of days 

{ 

in isn average week^ even if it may be just a few minutes stand- 
Ik 

ing in front of a store window. 
Item 18. Days per week one reads a newspaper. "Reading" included 
^rely glancing through. The number of newspapers were not 
counted, but rather the number of days some kind of contact was 
had. 

Item |9. Days in the past,month one has read a magazine. The clas- 
sification would include comic books, fotonovelas , and different 
kinds of pamphlets or tracts, and we would count having read the^ 
same thing twice if it had occurred on different days in the 
same month. - ' ' 

Item 20. Occasions during past month one has gone to a movie. In- 
cluded here would be the rare occasions when a movie, whether 
entertainment or didactic, was brought to the community from 
outside ( except when it had been brought by UCLA). 

11-31 



Items 16-20. People y/ere usually not sure of the exactness of their 
responses to these questions; in case of a vague or very general 
answer, e.g., "a few times/' "very little," an attempt would be 
made to reach as accurate an approximation as possible through 
more precise wording or requests to the participant to try to 
remember about how many times in each case. 

Item 21. Language spoken. The language used most often ,^ whether at 
home or tn the market, was to be indicated; the language or com; 
bination of languages the person .spoke best. Answers were ac- 
cepted as given, even when at variance with the facts (i.e., a 
person may speak or understand Quirhua yet deny it to a ques- 
tioner because of that language's lower status.) 

Item 22. Membership in a community organization. Any organization 

within the community that was formally constituted (i.e., not 

J" 

groups of friends or informal sports teams), regardless of its 
level of activity. 

Item 23. Position of leadership within a community organization. The 
positions of leadership were president* vice president, treasurer 
and secretary. General membership alone* would be marked as a 
negative response. * 

•Item 24. Religion professed. Responses were accepted as given. An-^ 
swers such as "liberal," if-unable to be pinned down to a speci- 
fic, recognized sect, would be. marked as "none." 

Draw-A-Man. The participant was asked to draw a compl ete human figure 
man. Initial refusal (a common phenomenon) wou,ld be met with as- 
surances, coaxing, support, snd even suggestions (as to subject, 
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not particular items withjji the drawing) from the field worker. 
One, two, and even three refusals would be argued, but after that 
we would simply drop the matter. If the person wanted to draw 
the figure from a magazine or newspaper, we would suggest he draw 
it from memory or imagijiatibn. 

Identification of letters, syllables, and individual w^ds. Eac/i sec- 
tion was done one at a tim6, the field worker askinVfor the ixlen- 
tification of individual items from an array of pos^sibilities. 
.Reading of sentences. The field worker followed each reading closely, 
identifying those words read correctly. 

Writing of participant's fiame, individual words, and sentences. Words 
were counted individually on each item (there were six); for this 
item "correct" was anything that was legible and that, when read 
phonetically, could be understood within, the context of the sen- 

, . tence. -This allowed a-fairly wide leeway in terms of missing 
letters, missing or wrong accents, lack- of capitalization, etc. 
Since we were measuring functional literacy, what could be read 
and understood by an average Spanish speaker was accepted as cer- 
tainly functional. 

Since there were often differences of opinion as to what was 
legible or understandable, the final .point score on each item was 
determined by the coordinator and the administrator. 

Identification of numbers^. These four items were done one by one, 
the field worker reading the individual numbers^(each item has 
two to be identified) and the participant identifying them with 
a mark. 
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Orally-applied math (addition, subtraction, multiplication) problems. 
EachK^oblem was given slowly, clearly, and one at a time. The , 
participant was reminded not to hurry and to think carefully. < 
The problem was repeated if so requested. The field worker was 
allowed to encouW-and assure the participant, but not to help ^ 
him in any way. Participants were allowed to count on their 
fingers, but not to wri^e down the problem. 

Written math problems. To be done by each participant^orking alone. 

« 

The field worker was to watch the process, reminding the partici- 
pant to be careful, ta read the problems carefully and to take 
all the time he needed; encouragement to continue despite diffi- 
culties was allowed but no actual help, or advice. If directly 
asked about a certain problem, the field worker was to either re- 
fuse to help or answer only very generally (e.g., "Where does one 
begin with that kind of problem?" "What is. this number? Does it 
go there?" "What does this sign mean?" etc.). 
Critical Consciousness'. This was by. far the most difficult section 
to standardize across field worker and individual participants. - 
Each question was to be ask-ed a first time as it was printed on 
the instrument; if not clear, it would be asked again, more slowly, 
using "replacement" words for certain phrases we had had difficulti 
with in the field testing (these "replacement" words were printed 
above their respective phrases and thus standardized). 
/ The problem areas were tjw^ First, if the question was still 
not understood after the s^ond reading, the field worker would 
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have to adjust it to fit the conceptual level of the interviewee, 
while trying to retain its essential thrust. Second, the variety, 
length, and vagueness of most of the respondents' initial answers 
defied the neat numerical codification appearing on the instrument. 
The interviewer would then have to question further the response, 
clarifying it or trying to narrow it down to fit the possible 
codes— at the same time making sure it was an honest answer, one 
that did indeed express the participant's feeling on the particu- 
lar issue. 

When the literacy and numerical skills tests were expanded 
(Version 2) in Decefnber, 1974, with the addition of more diffi- 
cult sentences to be read and written, ^this questionnaire pro- 
cedure remained the same. 

I 
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Work Diarf es 

' — t 

Any field investigation requires the keeping of , detailed Work diaries 
involving personal observations and analyses,, procedures followed and their 
apparent impact, Informal conversations, and any other kind of general in- 
formation that would shed light on the subject at hand. This is all the 
•more important in an investigation as complex as that carried out by UCLA 
In its evaluation of the educational materials. On the -one hand, we were • 
testing concrete materials that claimed to have 'both definite, measurable 
results and also results that were equally vital to identify though less 
tangible (e.g., changes in awareness, attitudes and behaviorsj. As a sec- 
ond factor, we were considering a series of processes (e.g., introduction 
to a rural community, establishment of a learning group, motivation towards 
a more dynamic outlook) tJiat also were claimed by UMas.s NFE and the other NFE 
projects (e.g., CEMA)- td-)iave both visible, measurable results and results 
that perhaps could never be completely identified. 

Within such a charge, the work diaries would Togically.be serving the 
function of identifying and analyzing, where possible, the innumerable fac- 
tors that would influence the use of materials and the processes. A second 
function would be to identify those less tangible results effected by 
those same materials and processes-. 

We needed a detailed mass of information on each community, and to ob- 
tain it. a detailed outline of a model work diary was developed. This was 
then brokei. down into three "sub-diaries," two of which were put on sepa»*ate, 
detailed forms and the third-the diary itself-which would merely follow a ^ 
structured set of guidelines for maki^'ng more general observations. 
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The introduction of the UCLA project into each rural community was 
identified as a key element in the total evaluation process, given the 
importance of local support and mutual trust for the project's success. 
A description of the introductory steps v/as to be the first part of the 
original field diaries, with the same information later pulled out to, form 
^^^separate, comprehensive record of'this phase- 

Our focus was on the two major variables in any such introduction: 

1. the' sector of the community with whom the original contacts are 
made, and 

' ' 2. the apparent "openness" of tbe community to this kind of outside 
project. 

A detailed description of the first steps was provided for each com- 
munity: contacts made outside the community (e.g., with the local priest 
or teniente politico ); and,, how this led to an identification of fcommunity 
leaders. Then we v/orked for a complete list of the people within the com- 
munity with whom "exploratory" talk^ were held-rhow they were contacted, if 
the visits were made by the field worker alone or accompanied by an outside 
contact, what aspects of the progranr were explained and to what degree, and 
what were the questions raised by the in-community contacts. 

The next step in our outlined procedures was to call a community meet- 
. ing in order to inform the people of the project's methods and goals and to 
ask for the participation Of interested residents. This step was broken 
down into its several components and described at length in the diary. We 
wante^ to know how the meetings were^ arranged (Who called them? Were they 
for the purpose of introducing the UCLA project, or was the field worker 
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able to "piggy-back," taking advantage of an already-scheduled community 
meeting?). Other questions were: How many meetings were eventually needed 
before- there was enough interest and support to begin the program, and how 
each meeting developed (what explanation was give^by the field worker? 
What questions or doubts were raised by the local residents?). Then we took 
a subjective look at the process followed to that point: What had been the 
reactions of the community leaders? of the residents? What were the £os- 
itive reactions towards the project and what were the negative ones? How 
did these help or hinder the formation of a learning group there? No less 
important, we asked that the field .worker describe his or her own feelings 
and reactions toward the community, the people, the process being followed. 

In short, how did the particular group of participants , come to be 
formed? From the demographic data obtained on the community and, in more 
detail, on each individual , we could see if the group was a microcosm of 
the entire community or if the project had been able to attract only cer- 
tain kinds of people or certain sub-gnoups within the community.- The in- 
formation on the introductory step^ vyould indicate the characteristics of " 
the group and why these kinds of people became interested in the program. 
Inferences drawn from information thus available would then indicate which 
additional steps (e.g., contacts outside the community leadership circle, 

* 

individual home visits, further information, etc.) ought to be taken to 
attract the kind of individuals at whom the UMass NFE materials are aimed. 

The second Key step was* the use of the educational materials, and a 
four-page form was developed to document their application in each session 
in each community. After noting specific data of the session-which game 

r 
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played, number of times it had b^feen played to that point, number of par- 
ticipants and number of observers in attendancl— the descriptive outline 
follows our procedural steps for use of the materials: 

1. -What was done to help form a "learning community" and did i,t accom- 
plish its goal (i.e., informal games, story-telling, etc., to re- 
lax the group)? 

2. a list of^all the objectives^ that were set for the game at t-his 
particular ses'sion, noting if they came directly from the parti- 
cipants or were suggested by the animador . 

3; a narrative description of exadtly how the game was introduced: 
rules and explanations by the animador . suggestions by the par- 
ticipants, questions that arose and how they were answered. 

4. any interesting points or problems that arose during the play of 
the game, and how they were developed. 

5. a description of the different ways the game was played in the*' 
course of the session." Were these changes suggested by the an,i- ' 
mador or by the participants? Were there any special observations 
made about these new ways of playing? more eff^cti^? problems? 

6. data on active participation in the game: how many of the matricu- 
lated participants played and how many merely observed? How lofig 
was the game' used (aside from introduction, question, etc.)? 

7. a detailed list of the most important ideas or 'concepts that came 
out during the. game. ' jin the case pf the literacy games, it would 
be a list of the" most important words.) These would be the ideas 
or words selected by the participants as most important afte,r the 
game had been concluded. * 

8. , how were each pf these ideas' or words 'develope^d in the "reflection" 

period? Wh*t questions were raised, where the discussion went and 
how it progressed (were the participants active or passive? to 
what extent did the animador have to intervene, suggest, and even 
direct?). 

9. returning to the objectives for the game that had been set at the 
beginning of the session,' how were they reflected upon? (i.e.j 
did the participants feel the objectives had been met, and to what 
degree? Had they been §er ^o high? too low? Could the objectives 
be met with this 'game at aVl?) Which of the objectives had not been 
satisfied by the end of the, sesston?- Why not? 



10. how were the conclusions reached by the group? Did the group 
commit itself to any concrete action or activity? How were these, 
conclusions reached through the ideas expressed in the games and 
discussions? --or were 'they? . | 

11. how was the n^t reunion planned? What did the g rou p suggest for 
the session (i.^. , informal activities^ new ways to play the game, ' 
etc.)? ^ ^ 

12. personal reactions of the animador / Was there a feeling of cooper- 
ation? Did the session -go smooth Vy? (Why or why not?) How does 
he or she feel with the group? 

13. general observations. These v/ould cover such things as feelings 
on the level of interest in the group, general progress of the 
program, utility of specific games vis-a-vis specific groups, etc. 

This form was constructed to be read individually as well as part of 
the Series in(^each community. We sought to detail the use of the games as 
individual events, itientifyjng factors at each step of the process (inter- 
vention of the animador, level of participation by the group, etc.) that 
bear on the ideas that emerge and how^the ideas are developed in the "re- 

flection^' stage. Then we wanted to have a sequential description of each 

/ 

non-formal game in the communities in which it was used in order to be able 
to trace key changes that might be expected to be observed over time: feel- 
ing of group identity and confidence, adaptability of the game to group needs 
and desires, conscious utilization of the game as a learner-directed educa- 
tional experience, ability to explore and expand upon the ideas that are 
sparked by the game. * . _ 

This exhaustive examination of each session is crucial in accounting 
for either cognitive or attitudinal changes that are observed in the quan- 
titative data, as no game was used in exactly the same manner in any two 
communities. 



Our third qualitative "contrdV on the field work experience was the 
more loosely structured work diary itself, meant to fill in the observa- 
tional gaps between the initial introduction and the game sessions and be- 
{tween the sessions and the community at large. The diary was to be subjec- 
tive, cdlling for ff self-exdiiii nation of the evalxfation process at work; 



how and where the instruments were applied, problems encountered with their 
use, reactions and questions of thJ (^is^mnu^^ participants involved, per- 
sonal feelings toward the work'itself andHhe community. Impossible\to 
quantify, the information was nevertheless vital since the evaluation had 
to deal with processes that influence the relative success of a non-formal 
education program— process^s^^th^ are implemented at a very peXsojial level. 
In short, how do certain kinds of people (e.g., a field v/orker) rmplement 
well-defined processes with a kind of reaction from rural peasants? 

In addition, we asked for subjective data on the communities themselves 
in order to obtain a fuller, more operational picture than thgt .presented 
in the demographic .profile. Specifically, what were the various facets of 
community life— economic level, geographic mobility, openness to outsiders, 
educational level, internal socio-economic differenced, relative dissatis- 
faction—that would influence the development of an educational program in 
that community? And as a "check" on these appraisals, we obtained a compar- 
ative look at each community in terms of its neighboring villages. (These 
forms appear in Appendix A.) ' • 

Uftess Impact Profile 

This is a ten-page form filled out for each of the eight rural commun- 
ities intervened by the Umass proiect iji which the evaluatior\ studied their 
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impact* These were completed in the course of the observational site visits 
and interviev/s in each of the villages. The profile gathers linf^Q]i^d;^n 
^ on the facilitators (thq^^community people trained ih.the use of educational 
games and leadership qualities) and on the various UMa>s non-formal mater- 
ials that were used in the community during and .after UMass NFE intervention* 

The fajcilitators . After detennining the length of direct UMass pro- 
ject actfvities (i.e., training- of facilitators through to the last site 
visit by project personnel) and the scope of community participation (age, 
sex of the participants in the learning group) > thjs profile asks about facili- 
tators themsel>res--age, education, occupation, their selection as facilitators, 
training received, amount of economic compensation (if any) received for 
their work. This information was obtained through personal in-depth inter- 
views with each (where p'ossiblfi). No difficulties v/ere encountered as most 
of the facilitators still l{ved in their communities and w6re quite willing 
to discuss the pai^j^Jjation^n Ind opinion of the UMass project. These 
data serves as quantifiable background to the interviews, allowing the evalu- 
ation to >assess the activities of the UMass project in personal terms. It 
is designed to aid in answering evaluation question #3 (the effectiveness 
of the UMass project in implementing its objectives) and #7 (teacher/facil- 
itator variables as related to the effectiveness of^e materials). 

The non- formal materials * The second half of the impact profile is 
an attempt to quantify the range and depth of the materials (games, foto- 
novelas , literacy methods, radio programs) used in each community* Thirty- 
five educational games, three "expressive methods" (posters, rubber stamps. 
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the Ashton-Warnef literacy method), the fotonovp^as and the radio programs 
had been identified in UMass project documents, reports and personal in- 
terviews as having been developed and field tested in rural Ecuador; each 
was included in the profile and its use (which games and how^many times) 
determined by interviews with the facilitators* These figures are exact 
to the extent possible (soipe games had^ not been played in more than two 
years) J but also serve as a kind of perceptual "check" in that their^ im- 
pact can be judged by how well they are remembered by the facilitators* 
This information aids in answering evaluation question #3 {the effective- 

't 

ness of Ut4ass project implementation) and, to a somewha^ lesser degree, 
#5 (the characteristics of the more and less effective non-formal mater- 
ials). The impact profile form appears in Appendix A. 

Annex to the individuctl test in the UMass sample > In addition to the 
test instrument given to the UMass NFE participants (same one used in the 
evaluation and "control" communities), each participant in the UMass im- 
pacted communities was interviewed on a standard format concerning their 
personal perceptions of the non-formal materials they had used. They were 
asked to name those games or materials that they could remember (or to de- 
scribe them, with the evaluation staff filling in its common name), and 
how long it had been. since they, had last attended a session where the 
materials had been used. Then the individual was asked to classify the 
games by which he or she had liked most and which they had liked least. 
To get at the question of the games' educational impact, we asked if '•the 
individual learned something *from them, and if so, what? (e.g., spelling, 
reading, math, about life, and combinations of these). To measure the 

*■ 
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Impact on tfie person himself, we asked the participant to judge the games 
in general according to their effect (great, some* little, none) on his 
or her personal life, on their family, in his or her work and economic 
life, and in the community at large. 

This information, though quantifiable, is taken more as an indication 
of the UMass project's effect at the individual level. We wanted to know 
how people not trained in the materials or the project goals viewed the 
games and the processes followed. In short, had they had an impact, ac- 
cording to the individual, in fundamental skill acquisition, on attitudes 
toward self and community? The answers to this would help us in answering 
evaluation question #3 (effectiveness of the UMass project implementation) 
This annex appears in Appendix 
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STAFFING 

In this section, the staffing of the proj^ will b^e discussed. 
Procedures for selection of field workers and ^thei}) training constitute 
the major .portion of this section. A discussion of criteria for 
sele^cting a fiield coordinator is also provided. 

Selection of the Field Workers '("Animadores") , 

With a fairly clear idea of the kind of work that would be involved 
♦ 

in gathering the data for the evaluation (physical and social setting, 
kinds of people to be worked with and on what.basis, type and range of 
instruments to be used), the care with wh4ch the implementers— the field 
workers ( " an i madores " ) —we re chosen was of crucial importance. 

Since the evaluation design .did not call for the use of people from 
the community itself as the group leaders, but rather someone from out- 
side who would be the group "anim'ador" (animator) in several communities, 
a markedly different set of criteria than that developed by the UMass 
NFE Project for their facilitators" had to be drawn up. 

On the assumption that there is a noticeable cultural difference 
between coastal and mountain Ecuadcrr (i.e., linguistic idi^m, personal 
style, mode of dress', perceptiort of self and outsiders), and since the 
animador was not to-be from the community or possibly, eveo the same 
area, v/e ppstulated that he or she should at least be from the sam^~geu=- 
graphical region— Coast or Sierra—as the commmities in which they 
would work in order to reduce the chances of "cultural static" affecting 
the outcomes of the evaluated materials. In instances where field 
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workers from the same region were not available, we felt it urgent that 
the ^Id worker have -extensive work experience in that area. 

A second criterion used in selection was work experience in rural 
connmunities; a familiarity with the wey people in rural areas work, 
act, and think was deemed vital, given the nature of the evaluation 
/i.e., long periods of time spent in small villages and qualitative 
judgments that would have to be based on an understanding of the milieu). 
But we were looking for more than just experience; we had to probe for 
the personal qualities that a person had brought to a previous experience 
and would have to bring to this one. Was the prospective field worker 
disposed toUravel long distances under less than ideal conditions, 
spend extende^eriods of time in small, isolated villages, work long 
hours at night and on week-ends? And even beyond that, did the person 
have, the ability, the desire, to establish relationships of equality 
with villagers, to truly share in the rural life style? High standsrds 
indeed, and difficult to answer "yes" or "no" for any individual. 

Our task in the rural areas was dual, to inplenient a non-formal 
adult education program and, at the same time, to evaluate it. The 
skills discussed heretofore related to that first task; the second one 
called for much more specific skills since it dealt with the careful 
documentation and analysis of, the processes being implemented. Thus, v/e 
set down as a criterion for our animadores previous experience in some 
kind af social investigation. First and foremost, we wanted a respon- 
sible person--one who could follcw the processes and the work calendar 
without the need for constant supervision and\:orrection. Secondly, 
it meant finding people with a discerning eye; people who could sense 
feelings in another person, make -sense out of the subtle nuances of 
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daily life in a village— ancj report these observations ,in a clear, 
concise manner, (After sonie consideration, we discarded the idea of 
setting an educational criterion [i.e., a certain level of schooling], 
deciding that it was relatively unimportant compared to some of the 
dther attributes under consideration,) 

Our search began,, as with the. rural communities, in Quito—inter- 
viewing people from AID, SEV, CEMA, UMass, and the Ecuadorian uriiver- 
si ties to first compile their recommendations into a list of field 
worker candidates. This process alone was enlightening; Ecuador, des~ 
pite its overwhelmingly rural population, offers few institutional 
'means by which young people from urban areas or the universities can 
work in the rural environment. Even fewer are the opporturriiies^fo-p- — 
work in social investigation, since private institutions are rare and 
the government as yet has taken little initiative in this area. Not < 
surprisingly, then, we would have to Icwer our sights considerably in 
the selection of field workers, choosing those candidates v;ho were par- 
ticularly strong in one or more areas while revising and intensifying our 
plans for "their training, 
/ The list of candidates was narrov/ed to the ten most qualified; none 

met all the criteria, but all had experience that would be valuable to 
the evaluation. Each was interviewed individually by the project direc- 
tor and the Ecuadorian technical advisor witTi the emphasis on open-ended 
questions designed to probe their desire to work and their originality 
in handling difficult situations. 

The five-member team chosen presented an excellent cross-section of 
talents and backgrounds: tv/o women and three men; two with experience 
on the Coast and three in the Sierra; one campesino, two SEV ( Servicio 
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Ecuatoriano de Voluntarios ) volunteers, and tv/o with social investiga- . 
tion backgrounds. The training, detailed in other sections, was. desired 
to give a theoretical framework to the evaluation effort and to specifi- 
cally^ll in those skill areas that were so necessary; in short, to 
prescribe and standardize the Wc^y in which each would carry out his or 
her work In the communities. 

The actual demands of the evaluation effort proved to be much, 
much greater than we had anticipated. Physically it was draining; long 
days of interviewing, which involved a tremendous amount of legv^ork, 
follor/ed by evening game sessions, all under comparatively primitive 
conditions of hygiene and alimentation. Coupled with the weight of 
working alone (except for occasional visits from the supervisors), the 
field workers began to feel strained and edgy. The other major aspect 
of the work— carefully controlling implementations and evaluations and 
fully documenting all activities-was the cause of further problems with 
the staff. A few were simply not convinced that all of the control and 
documentation were of any real importance and did them only under duress, 
while others were unable to do an adequate job for lack of skill in 
observing and writing. ^^^^ 

The pressure, then, was being felt from both sides: the demands of 
working in rural areas and the demands of a thorough evaluation. Three 
of the original team eventually left the project, unable to come to 

•'5 

grips with its multiple requirements. It was, at the time, a serious 
blow, but it forced the'project and its staff to re-examine ourselves in 
light of four months of di re ct- experience. 

An intensive review of all of our procedures was undertaken: criteria 
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for selection of conmunities and field workers, procedures to be fol- 
lowed in' the comnunities , the 'data of the ev^iluation, and the super- 
visory effort. In the end, we felt it to be a question of people- 
people who enjoyed working in the countryside and got a deep sense of 
personal satisfaction from it, people who were mature enough and honest 
enough to de'al with problems in an open, receptive way. We had found 
that previous experience was of little concern, but rather the most 
relevant factor was the att(ftude Vith which one approached the work; 
the rest was a question of irainin) 

In selecting new field Wbckars^en, we scrutinized much more 
closely the individual's personality: motivations, inner strengths, 
maturiity. Work experience was taken into account, but only, as an indi-, 
cation of where we o^to put the emphasis in the subsequent training. 
The four people wejipose to augment the team were, again, a mixture of 
origins, backgrounds, exlucation, and experience-but this time selected 
on the basis of>GS«rdered criteria and, most importantly our own 
scorching experience. Jhe quality of their work in the fol^lcwing months 
proved the correctness of our altered focus. 

Coordinator ' 

The position of "cQordinator" was a key element of the evaluation's- 
field design.- For a variety of important reasons (the physical disper- 
sion of the field team,. the physical and cultural conditions In which^ 
they would be working, the importance of accurate data collection, etc., 
there was the need for a person-preferably Ecuadorian-who could super- 
vise the workers in the evaluation sites, coprdinating andjtandardizing 
their efforts. In- addition, the coordinator was to help with thej^rain- 
ing of the animadores , and to contribute to the analysis of the^ 
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non-quantitative data at the of the project. 

•» 

The criteria were stiff, befitting the demanding work th^it would 
be involved: - • " 

1) experience in educational evaluation in rural areas of 
Ecuador. 

2) * experience in supervising the, work of others. 

3) ability to organize data to. answer evc^uatlon questions. 

4) ability a^d desire to travel and to spend prolonged periods 
of time in isolated rural areas. 

The selection process began .^with interviews with personnel of AID 
and of Ecuadorian institutions involved in either evaluation or rural pro- 
grams. Each of the candidates gleaned* from those contacts was then inter- 
viewed at length by the project director. 



.Srta. Pilar Nunez was selected as the evaluation coordinator, comin 



.to the project with the highest recommendations from AID Mission, from Dr. 
Donald Swanson, head of an independerft evaluation group in Quito, and also^ 
from the UMass NFE project staff. 

Training of Field Staff 

^ ■» 

The original five-person evaluation field team was trained in Quito 
during the first three weeks of September, 1974. Training ^essions for 
new field workers were shorter, reflecting the experience Kif\he first 
several months in which we were able to identify those aspects of the work 
thaVwePj^-fliftSt difficult t^rasp and therefore needed greater emphasis. 
These ses'sions were held during the last week in November, 1974, and in 
mid-January, 1975. All thref^ssions were immediately followed by con- 
centrated supervision in tHe fielld. 
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The training procedures are discussed here only in brief outline 
form; a more detailed description of the specific procedures and the 
weight given to each can be found in the separate discussions of evalua- 
tion instruments, procedures for introduction to the rural community, and 
utilization of the UMass materials. 

Each training session began with the theoretical background of the 
evaluation design: what is_ evaluation (as opposed to investigation or 
community development), what are the questions it tries to answer, and 
how is that information gathered. This explanation was^l^^ays a major 
component in the training; there has been relatively li tile -evaluative 
experience in Ecuador and the demands involved (in terms qf scope and 
quality) had to be made very clear to people whose experiences had been , 



in other fields. (We found this to be especially true in the case of 
those workers y/ith previous rural community experience; confident in one^ 
aspect of the project—imptlementation of material s--they confounded it 
with the other aspect— evaluatten of those materials)-. 

. The theory was closely followed by the concrete case of this parti- 
cular evaluation— its procedures and instruments. Each step in the de- 
sign wa^^roken down into its various components and thoroughly discussed 
in terms of why^ it was important and how to carry it out: the measuring 
instruments, the commutiity and game processes, the accurate reporting of ^ 
non-quantitative data. 

The first session (September 1974) included a supervised field test 
in a rural community nean Quito; subsequent sessions (November 1974, Janu- 
ary 1975) also had field practices, but these were held in^the actual 
evaluation communities already being impacted. 

. 6G 
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The UMass NFE educational materials were introduced to the September 
group by UMass personnel, who not oYrly gave the background of their pro- 
ject but ijnstructed the team in the manipulation of the games as well. 
The short, ^ne day "courses" were extended and run by the UCLA evaluation 
staff itself for the second and third training groups. 

, The September session also concentrated much of its attention on moti- 
vational training: inter-personal relations, personal goals, group dynamics 
working as a team, etc. While useful, we found the techniques were not 
directly applicabl-e to the task at hand; we had been trying to create a 
inotivational base instead of specifically selecting people that already 
had that foundation. This part of the training was. down-played in sub-" * 
sequent sessions, except where it might touch on actual experiences in the 

communities: techniques of soliciting answers to questions, making others 

feel important, running a game meeting ip a democratic way, etc. 

Standardization of procedures had been » major goal of the original 
September training, but had proven extremely difficult to attain under 
the conditions in which we worked. Nevertheless, as a fundamental con- 
sideration in any evaluation, it was treated at Idngth with the November 
and January field workers groups— not only whj^ it was so importa^Ht but also 
means for achieving it even under adverse, often confusing circumstances. 

^ ^ Training was not limited to a period before the field worker began 

in a community, but was followed-up by a series of group meetings (all the 
field workers and staff) to review the steps and the problems encountered. 
The early October meeting in Quito focused on a detailed review of the 
first week itr the field— every^ing was discussed and strategies designed 
to overcome unforseen circumstances (such as resistance to our questions 
being asked) at a time that the rather unpopular agricultural census was 
being taken. rsw 
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Thejecond general training meeting was held in Banos'in late October. 
At included practice through simulation community situations. 

Training for an evaluati^Dn such as this, where at least half .the pro- 
ject is implementation as well , is complex. The context of the work, iso- 
lated rural communities and the physical difficulties to be found there. • 
seems to require that an animador have previous experience in such commun- 
ities in order to minimize the "shock"; we found this to be, at best, only 
partially true. Rather, it was a matter of the kind of person behind the 
experiences people with great experience, but of a loose, undirected nature, 
can ill adapt their style to an evaluation study, while people with no ex- 
perience, but the inner strihgths and drives to sustain them in difficult 
situations, can be excellent field workers because they see -it as a re- 
warding experience. No training can impart the desire and willingness |o 
work within narrow boundaries under frustrating conditions; a training de- . 
sign can only build on an already existing structure. 

Ori'the other hand, intensive, repetitive training in the use of evalua- 
tion instruments is of crucial importance, regardless oi^ an individuals 
experience. Standardization in this area is absolutely necessary. 

Field supervision and "continuous follow-up training are the gther im- 
portant aspects of field worker training. They serve multiple purposes, 
personal as well as professional. The supervisors have a chance tp see 
how effective the training has been, experience for themselves the problems 
faced by the animadores, and then design new programs or altered procedures . 
to deal with the reality of the fielcl. The field'worker has the chance 
nd off" on his problems or complaints, getting individual support. 
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during the site visits and that of his .colleagues during team meetings. 
The keys to the succes? of such an interc^hange» however, are flexibility 
and maturity : the ability to deal with problems in an open, receptive . 
manner; adjusting procedures to meet those problems without sacrificing 
the needs of the evaluation at hand." * • 
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IMPLEMENTATION PRO CEDURES 

A number of steps were involved in implementing the procedures of this 
evaluation plan. Quite obviously,' great care had to be taken in the manner 
in which the communrties wer6 selected. Obtaining access and acceptance 
* into the conrounities was a vital next step. In this section, we have also 
described the specific procedures rejated to the introduction of the UMass 
NFE game materials. The final part of this section deals with the procedures 
used in selecting subjects both those who were participants in the games and 
those who were "controls". ^ ^ ^ 

Selecting Corrmuni ties ' for Implementing and Evaluating Non-Formal Education 
Materials . 

The evaluation design called for a limited replication of the UMass 
project procedures for introduc^on and use of educational materials. Since 
the evaluation project team was charged with the responsibility of determin- 
ing how mi/ch people learned and to what extent "critical consciousness" ^vas 
developed with the materials,- the first question was "What people? where?" 
We decided to evaluate the materials under the same physical and cultural 

conditions as they were used by the UMass project: in a series of mode^- 
- . ' ■ ' . 1 

ately small, rural communities in the highland pro\fcinces of "Chimborazo and 

Tungi^ahua and in the coastal province of Guayas. * 

UMass project documents .describing the f^cil itators projects in Tungu- 

rahua and Chimborazo ( A New Approach to Community Education by V. Ickis, 

-1972), in the Columbe region of Qhimborazo (" Columbe" by D. Andrade, 1973), 

and on the coast ("Colonche" by^B. Alcocer, 1974), specifying clearly the 
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criteria which the UHass project used Id selebtiJip the' communities to be im 
pacted; except for minor changes they\re the same criteria for all thngj 
areas, and were not substantially altered between 1972 and 1974. They are: 

1. of " mestizo " ethnicity because of the^inguistic and cuftural bar- 
riers existing in Indian communities (the exception was. the Columbe 
project, in which all the communities were to be Indian conmuni^ies) . 

2. accessible by heavy-duty jeep in almost any kind of weather. 

3. retatively "untouched" by outside institutions or programs. 

4. a high level of "ag iveness" (i.e., organized, active in civtc 
programs) and a high\evel of relative dissatisfaction with existing 
conditions. 

5. illiteracy considered by the local popirhp^ion tp, be a major problem 
of the community. ^ 

6. economic viability (i.e., a sound enough economic h^se to support 
proposed activities and changes). 

7. homogeneous with other communities in the same area (this in terms 
of ecological resources, major occupation of the residents, per 
capita income^ physical infrastructure, number qf inhabitants,/ 
percentage of illiterates, and ethnic composition). . 



The jefValuation project team adopted these criteria for the selection of 
coninm>lties for implementing the games but with minor adjustmeiits. We con- 
sidered the ethnicity of the community to be less important than the lan- 
guage spoken there, given the monolinguism- of . the education materials we 
were evaluating. This would in no way eliminate Indian or partially Quichua- 
speaking communities but rather the mono.Ujigual Quichua ones only. We 
agreed that the communities should have had as little organized impact from 
outside as possible, and in no case were they to have had any impact by the 
UMass program; for purposes of honest me^urement, the evaluation 
communities were to be "virgin." Finally, we put less emphasis 
on the presence of "aggressiveness" or of illiteracy as a 



o 71 

ERIC 



conscious conmunity problem, since we had no way to adequately identify them 
as such beforehand; rather, we sought communities that djd, have some kind of 
local organization(s) and that had little formal, schooling available for the 
residents. We added the criteria of "regular size," apparent in the UMass 
project's own selection of communities though not stated as a criteria; that 
is, communities of between 200 and -1 ,000 inhabi tants^-enough to support an 
adult education program. 

The rural communities selected by the present evaluators for introduc- 
ing the non-formal education games and concomitantly evaluating their ef- 
fects Vyere selected then, according to the following criteria: 

1 



2. 

3. 

4. 
5. 



communities which had not been impacted by the UMas^roj^. 

communities that were small and rural (approximately 500 to 2,500 
persons). 



communities not consisting of migrants (since the evaluation calls 
for repeated participation in the treatment and repeated measures). 

communities which were mostly Spanish-speaking. 

communities similar to those'chosen by the UMass project for impact 
(similar in terms of economic development, size, geographical area, 
type of economy," social and educational development). 

Our search for communiti'es in which to implement and evalua^jj^e the se- 
lected non-formal educational materials began by identifying the UMass-im- 
pacted areas within each, of the three provinces in order to select communi- 
ties that were in areas physicaj-ly and socio-economically similar to the 
UMass-impacted communitiesf our intent, of course, was a replication that 
would be as accurate:^as possible. 

Then, armed with detailed maps of these regions, two evaluation team 
members (Dr. Laosa and Mr. White) set Qut to interview a variety of people 
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who were familiar with the various zones and could indicate possible communi- 
ties: UMass project personnel and field workers, Servicio Ecuatoriano de 
Voluntarios (SEV) personnel, people who had worked w.Uh the original Centro 
de Motivacion y Asesoria (CEMA) non-formal education project, and UMass NFE 
community facilitators. In each case the nature and purpose of the evalua- 
tion was spelled out, as welL,as the characteristics of the villages we were 
looking for; in turn, we would solicit a list of communities where we might 
work, as well as accompanym^t (if and when possible) to those conmunities 
to make critical contact withxhe leaders. From these interviews the eval- 
uation team obtained a list of between ten and fifteen possible evaluation 
sites in each of the three provinces, ^ 

The next step was to visit the communities again personnally in order 
to verify more closely the extent to which they met selection criteria and 
to make contact with the local authorities. This was carried out by the 
Ecuadorian evaluation field workers themselves, following careful instruc- 
tions by the director. In each of the regions they visited— the Pfllaro 
and Quero areas of Tungurahua, several areas in Chimborazo province, and 
the Santa Elena peninsula of Guayas— they visited each of the coirmunities 
on the initial tentative list, filling put a brief demographic profile that 
indicated the degree to which they met the evaluation criteria; in addition, 
they met with the local authorities and community leaders to gauge their 
receptiveness to the program, A secondary go^l of the visits was to iden- 
tify and describe, either by a direct visit or interviews with local leaders 
of nearby communities, other possible communities in the same region. 

The profiles and the reactions of the field workers were brought back 
to the Quito office and reviewed. From these discussions the evaluation 
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team selected five communities—one for earch field worker—that would defi- 
nitely fe^mpacted by the UCLA implementation/evaluation in the first week 
of field work* Those that had not been rejected outright were put on a 
"pre-selection" list; they would be explored more carefully in the following 
weeks and would eventually serve as the remaining evaluation sites. 

These same careful steps were followed throughout the year as more com- 
munities were added to the UCLA implementation/evaluation. Our experience 
often paralleled that of UMass project personnel, in that we continually 
found it impossible to locate communities that met aJl the criteria for 
ideal impact areas. Rather, we chose those that, within given areas, met 
most of them. 

We placed emphasis throughout on the personal visits made to the com- ^ 
muni ties before their definite selection, and the wisdpm of this was con- 
. firmed by the later field experience. The verificai^foiT^ required 
by our demographic criteria was important, but eve/i more so were the contacts 
made with community leaders and the field workers* reading of .the '^atmosphere" 
for the success of the program. The feelings and impressions— so often in- 
tangible— taken from a community were ususally as reliable indicators of 
that community's possibilities as we»:e_t|Te specific, identificable charac- 
teristics. » ^ 

* 

Within the acceptable range of variance from the selection criteria, 
we ^encountered a great number of possible evaluation communities, some 25 
of which we impacted to one extent or another. Some were unsuccessful from 
the outset, while others presented a variety of problems in the implementa- 
tion of the program and thereafter. Each of the communities is described 
in Appendix c. . 
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Introduction to the Communities 

After identifying a possible area for UCLA impact, or even possi- 
bilities among individual communities, the field worker (sometimes, accom- 
panied by the supervisory staff) would make initial contacts with formal 
authorities outside the community itself: the parish priest and the ten- 
iente poli'tico (the area-wide political authority, apppinted by the pro- 
vincial government), and the school teacher. The purpo<^e of these contacts 
was ipiJltiple: 

1) to inform them of our presence, and to outline our methods, 
goals, scope and institutional links. 

2) to seek information concerning injiividual communities tw ^ 
the area--communities that woulcr be interested in su'cTslKN 
program, the level of cooperatwn we might expect there, ), 

4 and the names of community leaprs (informal as well as ( 
formal) with whom we could continue the conversations. 

3) to seek their support in the program--a passive support 
at least, and an active one if possible (i.e., accompani- 
ment, of the field WQHcer>feo the community and in making 
the first contacts there.) 

i. 

These contacts were of the greatest importance. Local authorities 

4 

generally have a great influence even at the community level; their sup- 
port—or at least their acquiescence--is a '*sine qua non" for success- 
ful entry. They are, as well, jealous of the influence they wield and 
must be kept informed— and reassured—about the nature of programs that 
operate in their areas. 

Next was the contact made with the community itself, through visits 
to its leaders, formal (president^ of the cabildo , leaders of 1;he different 
organization) as well as the informal (the identifiable "movers" who may 
or may not ho^d an official position). Again, a detailed explanation was 
given of the UCLA program— its nat.ure and scope, and what would be 
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needed in the way of local participation anti support. These contacts might 

'J 



be made individualiy with the different leaders, or through a meeting with 
the cabildo as a whole, or other local organizations. Questions were ans 



wered and, hopefully, their support enlisted. 

ft A general meeting to inform the residents of the communi'ty was then 
called; it could be specifically to introduce the UCLA program, in'wh'ich * 
case visits would be made by the field worker and, ideally, one oKmore 
local authorities. These visits were made to individual homes to inv\te 
the people to attend, or sometimes they were "piggy-backed" on an already 
scheduled meeting of some kind (e.g., the town council, the sports club; 
the Padres de Familia— P.T.A. ) . 

The steps to be taken at these information meetings were detailed in 
outlines developed in September, and were to be followed as closely as pos- 
sible. A good deal of information had to?be imparted, which in turn would 
raise questions and doubts on the part of the residents that would have to 
be clarified. If great care was not taken here, there would be a recurring 
problem of uncertainties and possibly "suspicion throughout the course of 
the program. ^ 

First, the animador was to "recognize" the local authorities, thahk- 
ing^iem-foiMibeir help. Next would be a detailed explanation of th^UCLA , 
program: what the participants could expect from the project and what the 
project would expect from them. Emphasis was placed in the mutuality of 
>:esponsibili.ty,. and the"fact that the project did not incltjde material aid 
to the community but rather was strictl;^ educational , At this point t^^s- 
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tions were solicited— clarifications, expectations, doubts, even objections; 
each was treated as important and answered fully. If the field worker de- 
termined that another meeting would be necessary to ensure local partici- 
pation, this would again be arranged through the community leaders. If not, 
the meeting* proceeded to the next step. 

A "work calendar" would be established with the interested group: a 
day and, time most convenient to theili for the weekly sessions, and the set- 
ting asicj^ of several days before hand during which the animador would be 
in the community to take the pre- test. Arrangements would also be made for 
the site of the sessions, again making it a decision to be made by the group 
and not the field worker. 

Having outlined the Characteristics of j:he people with whom .the pro- 
ject was most interested in working (and reassuring the group that anyone 
could attend the sessions, indeed that we welcomed, the participation of 
everyone), the field worker would then "enroll" the formal participants 
(i.e., taking down Meir name and arranging a time and place for the appli- 
cation of the pre-test). If an adequate number of participants could not 
be enrolled at one meeting, the help of the others was enlisted to notify , 
neighbors or friends. 

Besides the step-by-step development of the introductory sessions, 
they were to be as informal as possible; the animador was an informant, 
a community resource, and not a traditional-style teacher ftiere to impart 
knowledge and direction. We emphasized the atmosphere ( ambiente) of these 
first meetings since they were to serve as an indication of how the actual 
game sessions would be carried out. 
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After the initial introduction to the community (described in another 
section), the evaluation field worker would return to that community on a 
weekly basis. Each non-formal education game was played five times (led 
by the evaluation field worker) in each community,' one game session per 

week. Thus, in each of 11 cormunities two games were played in sequence 

^ 

for a total in each. community of ten game sessions over a chronological 
time span of ten weeks; in four communities one game was played for a total 
of five game sessions over a chronological time span of five weeks. 

Individuals participating as subjects were individually administered 
,a "pre-test" before the first session of the first game in the sequence 
for his/her community; a "second^ test" was given by the evaluation field 
worker after the first game was played five times consecutively; and a 
"third test" was given after the second gar^ in the sequence had been played 
five times, > ^ 

Selecting the Subjlct^ (Participants and Controls ) 

A sample of 20 people per community was decided upon for irrelusion 
in a "treatment" group in each community. This number was selected in 
order to account for any attrition occurring over the ten week period and 
still leave a reasonable number of participants. Even with an attrition 
rate of 40% per community, one would be left with an of 12 people per 
community for the "third test." This would still allow statist^ical treat- 
ment of the data. Attrition data was also collected (i.e., reasons for 
attrition and characteristics of drop-outs). Q 

Any person who wished to participate in the games" was welcome. How- 
ever, only those selected as "subjects" received the repeated testing. The 

80 , • 

11-63 



same subjects who tested at time #1 were also given test^#2 and #3. It 
was our plan that the same evaluation field worker who introduced the pro- 
cedure into a community would also perform the rest of the procedures for 
that community in order to provide continuity. In some cases, supervisors 
assisted the- field workers in kdmTm^^ the instruments, addition 

to the individual testing and the Demographic Community Profile, each 
evaluation field worker kept a structured field diary in which he/she docu- 
mented the exact procedures followed in the implementation of the games 
and any other relevailt events or changes in the community or in individuals. 

In any design involving repeated testing, it is always appropriate 
to compare results of the groups under study with results on the same mea- 
sures "obtained by a "control" group. The control group was subject to the 
same measurements as the treatment groups, but in contrast to the treatment 
groups, the control groups did not receive the treatment. 

For purposes of obtaining a control group, a sample of betwee n 20 and 
.* 

30 persons, of similar characteristics and from similar communities as the 
ones receiving the treatments were selected. The control or comparison 
group was administered the same individual measures as the treatment groups, 
.following the same chronological sequence and time span intervening between 
measures^ The control group>Jowever,- did not participate in the non-formal 
'education game. 
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Pr ocedures Followed by the Evaluation Staff in Conducting- the Non-Formal 
• V 

Education Sessions . 

As is apparent from a close reading of the UMass project-produced 
technical notes and field reports concerning their educational materials, 
as well as from interviews with UMass prbject personnel", the games were not 
con^ved to be isolated objects that, of and by themselves, would produce 
the' cognitive and attitudinal objectives. They are experimental and were 
•seen as being at maximum benefit when used in certain contexts and following 
certain additional activities and procedures. For the conduct of this evalu- 
ation study, it was necessary to a^in a greater degree of standardization 
in the implementation of games and sequences. 

The methodology developed by the UCLA evaluators for the use of the 
materials was based in equal parts on a content analysis of each of the 
games to be employed, a list of the UMass project stated objectives for 
ch game, and a structuring of the different methods and steps used by 
ss in their own use of the games. We needed a method that would: 

1. fully explore the range of cognitive and attitudinal possibilities 
inherent in each game. 

2. replicate to the greatest extent possible the UMass implementation 
of the game sessions, and 

3. beS^^lied in a standard manner in all the communities ia order to 
draw g^eralizatile conclusions about both the material and the^ 
manner in which it is used. • . 

A five-step game session outline was then developed., 

1. Establishing rapport. - x ' , 

By means of an informal conversation about , community. 1 ife, local leg- 
ends, local problems, ^r by a more structured approach using lotal games, 
jokes, music programs featuring the participants, the animador would begin,, 
each game session in an atmosphere of informality and 'enjoyment. With every 
one— including the animador —bringing individuaK^periences , preoccupations 
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and needs- to the session, tfe^e was the. initial necessity of bringing the 
group together as a cpnscfous uni't; at first as a means of relaxing people 
who often arrived, tired from a day's work, and then as an activity that * 
-H/ould .create a shared experience. There wfere no set rules as to how this 
would proceed; the field worker could suggest different activities or 
Inore commonly, pick up on themes suggested by the participants. This' 
way, the informality of the session was established from the outse-t- the 
session is enjoyable^. it is created by and belongs- to the group, and the 
ammador IS there as a resoO^ce and not as a teacher. More importantly, 
It created the nepessary atmS^phere of "group-ness" which the full benefit 
of the games purportedly depends. • , ucmcul 

»> 

' ■ 2. Setting learnij^g. objectives. ^ 

Since the Uflass project materials were not intended to be used "on" 
JP|p:ci_par>t5but "fay" them_, the laying groups must feel -that they 

-have controTSver them, that the materials fit the Participants and nht 



.^vt; Lunuroi o^er inem, that the materials fit the participants and not 
the reverse. Before beginning play^ the animador would elicit from the 
group Ideas -on what things 'they would like to learn with the game at thi^ 
particular session. When a number of possibilities had been aired, the 
group would proceed to an ordering of possibilities, setting priorities • 
based on the importance to .them of the suggested learning goals. This was 
a relatively .easy- step. to eJ^ecute, especially when the participants ibe- 
came more familiar with the materials, but^it served several important 
Ihn*^!'°"^- l^^ participant, it established a feeling of ownership over 
.the games; forced^^to reflect on his^true learning needs, he/she coSld play 
•IfLfT concrete content and at the same time set the rules and 

steps by which the game would be used toward meeting those needs For the 
Mimador, the stated objectives gave clues as to the felt needs of the 

tho'se'eSd. 'a? l°''lh^ -^;^^^!' ^° '^^'"'^ ^'^^ °^ ^f^^ ^"si°" towards-. 

.P^^fnn ^n'lH 1 °? l^^ objectives servecT as a scale against which the 
sess on could be informally judge'd.by .the group; had the objectives been 
met; and if not, why not? Hhi.ch o bjectives were met and which were not' 
This usually provided materUPf^r group discussions. ^ 

3; Introduction and use of ttie materials. 

-i.nrA»^!lt>i7-i?*?P.'" game's learning process is the "concrete experi- 
4p&.n- 5."'^ ^3ter, serve as..tft6 .fa^sis o/ reflection at the end of the 
f uid'b. tL experience-: 15, ofV^cours"e, the game its.elf as t 

-Ja-m^% ft^! fr'^'^^f Jt'^f P^ceded^by a brief explanation of the 
SoF!3*Lf J.^'^^f ^""'^ "^J'y ''"^^■^ P'-evious session, 

nBking;sure;tMt the';ruUs are understooti. At this point suggestions con- 
cerning ways to-us€ the game are solicited trom the group--wa?s of pl-aJiSg 
tZ?ll'\'r/J'''^^ P^rticipants..-etc. Focus if againlen-^' 

t^ed on what the participants themselves, want* to get out of the session' 



The game proceeds as much as possible under the direction of the group 
or leaders from within the ajpoup- The animador facilitates play by answer- 
ing questions or perhaps stopping play momentarily to discuss with the group 
a point that has come up during the game; he does everything possible to 
avoid taking on the directive role. Play continues until the group shows 
signs of tiring or running out of ways to p>ay, ^'-^ 

Th^e second step, "reflection," is by far the mqst important of any 
game session. Through an increasingly profound process of questioning, 
analysis and 'further questioning, the "concrete experience" of the game is 
applied^^o the reality of the participants' daily lives. The animador take's 
a more active role during this step, beginning the questioning process at 
a relatively simple level and then sharpening it at each succeeding level. 
It is an exceedingly fine line to tread: guiding the reflection process 
but only in general terms since the ideas and opinions should come from 
the group, not from the animador. 



The "reflection" process, begins with a comprehensive review of what^ 
was seen to happen during the game: how did people act--together or in- 
dependently? How did they feel about t>re experience--good? bad? confused? 
Were there any problems in playing the game? Then the discussion is nar- 
rowed a bit: what v/ere the most striking things about the play just con- 
cluded? Which aspects seemed most important? Using' the infarmation gener- 
ated from these questions as a base, the educational possibilities of the 
game are discussed: of these important aspects of the game we have identi- 
fied, from which did we actually learn something, either concrete (numbers, 
for example) or in terms of perception (roles that were played, person 
^feelings)? Finally, are there different ways the game could be utilized 
^0 reach the same goals? Or how might it be used to reach different goals? 

At this point there is time for "conceptualization," or a clarifica- 
lion of what has been learned up to then. First, the ideas that have been 
di-scyssed are put in order by the participants, in terms of which are most 
important and why, "^^n, they are expanded upon, if necessary. That is, 
a^question may arise asrssthe actual legal requirements for the formation 
off a cooperative, or a question concerning correct spelling, .or one about 
mathematics. The animador acts as a resource, clarifying these points. 

They are then discussed in terms of their practical application. 
Taking the list of topics that have emerged during that session, the ani- 
mador initiates a discussion of each, this time focusing on a comparison 
with the daily lives^of the participants. For example, if the lawyer in 
the Hacienda game played a significant role^ it is compared with the lived 
experiences of the group: 2I that how lawyers are? Are there good Uwyers^^ 
and bad ones? etc. If the topic is one of correct spelling, (e.g, , spring-" 
ing from one of the literacy games), the discussion would begin with people 
describing how and where they- use theirWiting skills, and if correct 
spelling really is important, ^ 

From there, generalizations are made about the experience) of the game 
and those of real life: wfv^ such a situation exists. This is the true 
"reflective" step in the process; parti ciparlts are asked to go beyond what 
'is seen and felt to an analysis of the forces (structures, habits, people) 
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that cause those situations. Thus, one's daily experience becomes subjective 
and not absolute; because the* causes can be identified, action can be planned* 

The proposed action begins with a discussion of how the participants 
would like to see these situations changed and, more concretely, the steps 
-they might take to bring about that change* ' The 'educational games are 
brought Hack underxdiscussion by returning to the objectives that had been 
set before play ana\those that bad been met, and applying what had specifi- 
cally been learned to the desired changes being talked about. That is, "of 
what practical application are the things we've learned here?" 

The action being discussed is formalized by having; the group set objec- 
tives and goals for itself, centering on the ideas that came out in, the 
game. If it was a cooperative, for example, the goal would be to find out 
more about how one is organized and run. It might be one of literacy, in ^ 
which case a goal would be set to practice their skills more often. It is 
important that the goals be attainable; at first glance they might appear 
absurdly simple, but at this stage it is engaging in the process (identify- 
ing problem areas, reducing them to manageable components, and setting goals 
for their resolution) that is as important as the result. ^ 

4. Planning the next session. 

Each game session is quite unique, despite the general outline followed 
at all of them; topics that come ^ out of the game will vary from one week 
to the next, as will the experiences brought by the participants. The ani- 
mador must take that into consideration when planning, a session, but he is 
by^ no means left without guideposts or "bridges." The experience of each 
session offers subjects for the next: problems in utilization of the games, 
learning objectives that .were set but not met, activiliy goals agreed upon 
by the participants. They s^rve to link one session wt\th the next, build- 
ing on what has gone before rather than seeing the games as isolated inci- 
dents. The planning is, then, a joint process, reflecting the "possession" 
by the participants and the facilitating role of the animador . The fgnner 
set the tone and objective, and the latter helps to structure them into 
concrete actions. 
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FIELD SUPERVISION AND QUALITY CQNTRQI 

The essential element in even the most well- conceived of plans is' the 
extent to which a system of monitoring and self correction are present. 
Project staff was mindful of this from the very beginning and initiated sys- 
tematic procedures for field supervision of staff and attempted to maintain 
and improve the quality of instruments utilized based upon field experiences. 
Field Supervision . 

^The actual field coordination began after the first week of activities , 
in the se)«reedconTiiunities. Each field worker was visited individually. 
Given the short time available in. each community, the coordinator would fol- 
low a rather set format for the visit: 

1. informal conversation with the field worker to discuss procedural 
or personal problems, concerns, etc. Each would be worked out 
within the limitations of the evaluation design. 

2. review of work diaries, field notes, and other evaluation instru- 
ments. They would be checked with the animador for comple^ness 
and depth. : ^ 




3. help with the data gathering- -application of the tests, filling in 
demographic information, informal and structured contacts with com- 
munity leaders and participants. 

4. .planning the animador' s activities. This included the preparation 
Of the written design for each game session as well as the activi- 
ties Ontroduction to communities, test-taking, flow of infomiation 
to the project office) to be accomplished in the succeeding weeks. 

5. assistance with and supervision of the game sessions. In aiding 

the ammad^r with the participant' groups (which often exceeded 30 

to 40),^ the coordinator was a>le to become more familiar with the 

reality of each community, ^a-Tter procedure if necessary to meet 

^ new developments, and assess the quality of game use by each ani- 
, mador. - 

6. review of procedures. This was the last step in each visit, done 
on an as-needed basis; the coordinator v/ould go over with the field 

..worker those procedures that, through her observ^ation, needed cor- 
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rection or reinforcement, suggesting ways to imp*;ove their work. 

^' ?ni"I?.T^^;L'"c """""^'^^tion Thr'ough the visits.' the coordina- 
tor also acted as a very personal link between the abstract goals 

?hp EL?h'^"'J'°" '^^^ immediate, personal experiences of 

the field workers. Often alo^e in the isolated communit es the 
ammador needed the reassurance and support that S?ta NJ^ez the 
coordinator, could bring, as well as the reminder that their'work 
wa| important and that they weren't alone or forgotten. BeLeen 
the team meetings, her visits served as communications link between 
in]inadores. relating common problems, bits of news, greel^ng?:^ 

The coordination effort, as originally designed, was of comparatively 
limited scope; we envisioned one person--the coordinator-to travel between 
four days and two weeks at a time, vis-iting each field worker once every two 
or three weeks. The rest of her time would be spent in collating the infor- 
mation gathered and organising it in the Quito offfx:e.- 

As time went on, however, we found important reasons for expanding that 



scope: 
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^' ^''^^^ ^'^^'^'^ esch anima^ • 

^ nr^^r'J'h'n'' ""'"^ °^ ^'^^ co^umtiiil7ithin each area The 
project had no access to a car or jeep, depending entirely on pub- 
lic transportation or a good pair of boots; this proved to be 

' lT7lT.lVnT''''''''' time-consuming, and uuL'L y impo ible 
as a means of covering all the communities. One person even if 
travel ing constantly and doing nothing else, could nSt 'hope to 
adequately meet the needs of the coordination design" 

2. depth .and scope of the site visits. Responsible for the complete 
accurate completion of all the evaluation procedures, the ?oSrdina- 

• Jjlir^^u'^''' *° ^P'"^' ' °^ mMmms, one day "?h each 

f eld worker; an adequate visiTWTd be for two days When mu??i 
plied by five fieW^orkers-and then again by the &to fSur com 
' munities in which each animador worked--the sheer quantUy of the 
■ work became overwhelmingT^ quannry or the 

3. field worker -personnel. We fouhd that several of the animadores " 

car'rv nT"^'rr"'°"' technical d if? ifSllfS^ 

^ fir7f ,1 Jt' assignments-often lonely and frustrated. In the 
•first few months they required concentrated attention. It was a 
heavy personal investment, made at the cost of assistance to the ^ 

in a sZnn.'h S"^^^'''^ ^'^^ '^'^ it would Result 

in a strengthened person and, hence, an improved performance . 
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Nevertheless in a few cases we had to teminate the service of 
inadequate field workers. 

> • 

It was the issue that forced the project staff to step back and seri- 
ously question the coordination effort. Was it necessary? How much of it, 
which facets of it, were most important? Was it physically possible? 

We were unanimous in concluding that it was. necessary— in fact, indis- 
pensable. We also agreed that every aspect of the coordination as it had 
.been outlined prior to the field work was also necessary; nevertheless, we 
ordered those points, according to the priority in terms of the evaluation's 
needs and, secondarily, in terms of the needs of the particular field work- 
ers. Finally, we concluded that it was physically impossible— for one per-, 
son. Therefore, the project'sN^ield administrator, Peter White, would take 
on part of the coordination work, following the same procedures as the Ecua- 
dorian coordinator. 

This decision raised. the question of "coimiuni cation": could 'a North 
American (Mr. White) take on this role vis-a-vis an Ecuadorian team working 
in small rural Ecuadorian villages? And what effect would it have on local 
.people? The answer, as it turned out, was a qualified "yes." It was ex- 
plored with each of the field workers in terms of the communities in which 
they were working. Where a problem might arise, the field supervision would 
be left to Srta. Nunez exclusively. Familiarity with the-instruments, a 
good command of Spanish, and increasing familiarity with the project communi- 
ties were sufficient to overcome the doubts aVout Mr. Whitens impact on the 

^local participant groups. Under the circumstances it was the best solution 

( / 

Wssible, and it did serve to maintain the high standards of the coordina- 
tion effort at the same time it allowed "for th-^t- effort to cover all the 
evaluation project's many facets.. 
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The key to. the effectiveness of the supervision/coordination was plan- 
ning— "coordination of the coordination." The progress of each animadbr in 
each Gonmunity was charted in the project office, as were the reports and 
impressions brought back by Srta, Nunez and Mr, White, This helped us keep, 
track of the project as a whole and also served to make the site visits: 

1. increasingly personalized- Specific problems or needs could be 
dealt with on an individual -basis. There were, of course, problems 
common to all the animadores but which were manifested in different 
ways; aware of them beforehand, aware, of things that were coming up 
in each community (game sessions, application of tests, etc) visits 
could be planned to coincide with these needs. 

2. sequential. We were able to assess the l^ork of the field workers 
over time and on a community- by- community basis. Visits would then 
be progressive rSther than repetitive. They would be scheduled not 
according to a rigid calendar, but according to necessity (or lack 
of it). 

The field supervision, then, had two ^ major functions: 

1, to ensure the quality of the data collection, and 

2, to continually balance. the essential needs of the evaluation and 
the reality of rural communities. 

In both functions the coordination proved to be not only vitally inipor- 
tant but very successful as v/eTl , 

Quality Control of the Demographic Profile- , 

The instrument v/as reviewed point by point with the field team before 
work in the cormiunities actually began and again after the fir^ week of 
field work. Questions were raised concerning. the scope of some of the data 
requested arrd the possible errors because of inaccurate information available 



from local leaders. These were discussed, and strategies were set for the 



interpretation of widely varying responses to the same questions in certain 
conmunities. y 
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Individual field supervision was also employed; additional contacts, 
interviews, an<i observations by the supervisory staff helped to clarify 
some data and e)4pand on others. 

The original investigation design called for the demographic profile to 
be completed prior to the .introduction of the educational materials in order 
to get a picture of the community prior to intervention. Because some of 
the important data were either unobtainable or unreliable after only a week's 
presence in the community, the game sessions went ahead on schedule while the 
'profile would be filled in as more accurate data could be determined. Like- 
wise, the instrument's second application (immediately upon completion of 
UCLA- intervention) was often delayed far the. same reasons. In all cases, 
however, the information recorded was adjusted (if necessary) to conform to 
the status of the community immediately prior to UCLA intervention and imme- 
diately after the project there was completed. 

There were instances where the information was not obtained because the 
regular sources were highly unreliable ^nd greatly at variance wijth each 
other; theseitems were, regrettably, left blank rather than filled injvi,th 
possibly misleading guesswork. 

Quality Control of the Individual Instrument Application 

When the field work began in late September, 19Z4, we were confident 
that uniformity had been achieved concerning the use and interpretation of 
the pre-test instrument. Because rural investig^at^Jn can be -fraug^ht witlj^ 
unforeseen problems, two mechanisms were devised whereby the standard Jta^tf on 
of application could be maintained: individual field supervision andTji- 
odic group sessions with the entire team. ^ ^ \„ : , 
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The first group session was held in Quito after one week of jn-communi.ty 
field work, with no supervisory control during that time. Each worker was 
asked to descrijje in detail his or her method of applying the instrument 
and, item by item,, identify difficuUies encountered (i.e., problems with 
interpretation 'of certain words). 

Among the many processes to be followed by the field workers— in,tro- . 
duction of self and project into the communities, individual interviews, 
game sessions, work diaries— we agreed that the accuracy and reliability' 
of the interviews were of primary importance. This, then, was the focus of 
the individual field co-ordination during the' second week in t'he communities. 

One aspect of this was direct supervision: observing the fieW^workers 
as they applied the instrument, gauging their style, attitude, and response 
vis-a-vis the interviewee. Observations, criticisms, suggestions, and en- 
couragement were made where applicable, always in .a private, relaxed setting. 
Another aspect was by example: The supervisor(s) would ap'ply the instru- 
ment personally, with the field wor1<er observing. This allowed the supervi- 
sor to experience the vagaries and difficulties as well. 
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The pre-test remained at the top of oiir "pnonty list" throughout 
October and'into the first part of November. A second group meeting was 
held at the end of October. Each and every questionnaire that had been 
completed to that tim'e wa^ reviewed by field v^orker and supervisor, cor- 
recting errors, filling in missing demographic data, and again emphasizing 
the importance of uniformity j'n its' application until agreement was reached 
on how to deal with them. 

The instrument application was again reviewed at the subsequent group 
; meetings (in December and February), and in all field visits by the two 
1 supervisors right to the end of the project. 

Quality Control of the Non-Formal Game Implementation . 

The use of the UMass educational games requires two basic skills: 
physical manipulation (implying familiarity with the game itself, an ability 
to explain it to others, and flexibility. in its use according to a partic- 
ular learning group), and secondly, ability to utilize the game in its larg- 
er, reflection-producing context (to be able to pick up on ideas, conduct 
discussions in a prod'uctive^. non-manipulative manner, and help others reach 
logical conclusions). ^ / 

Given the ambitious objectives of the games and the evaluation's inten- 
tion tomeasure them.accurately, both of the required skills are equally 
fundamental. In the three-week training session for field workers iiv^eotem- 
ber, 1974, the training in actual use of the materials was left to UM^ per- 
onnel, who conducted short sessions with each game. The UCLA evaluation 
consuH^nts, already familiar with the game, reinforced those sessions and 
at the^satne time, they began'to concentrate /on a second, more difficult skill 
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The theoretical aspects of the "learning circle" (concrete experience -> 
reflection -> conceptual'ization -->^practical application) were discussed 
at length with^^ the field team, emphasizing their importance in the context 
of the evaluation. They were then applied to the non-fomal games in an 
intensive, several-days practice, with each worker alternating in the role 
of animador and the' others acting as participants. The other field workers 
then acted as supervisors, along with the trainers-pointing out shortcom- 
ings, steps skip-p?^ possible alterations, strategies for dealing with un- 
foreseen problems in the" course of a real session. 

The importance of standardization in procedure was made-clear, as were 
the great differences in content that would be encountered in each coiinnunity. 

Subsequent training sessions for new field workers followed the same 
format, but with UCLA consultants and staff in charge of all aspects of 
training. The later sessions were strengthened by our field experiences, 
from which we had learned that both physical manipulation and discussion . 
leading were indeed equal factors in the success of the games. 

Ensuring skilled game utilization required two quality control strat- 
egie's: direct field supervisior) and observation, and periodic group meet- 
ings at which the field workers could share problems,, frustrations, or suc- 
cessful tactics. With respect to the supervisory staff (Mr. White,' Srta. 
Pilar Nuffez), the diver^'ty of experiences during the first month of field 
activity put a severe strain" on them. Each learning group was unique, and 
the outlined procedures for game sessions were often being greatly modified 
by the anim^dores in an effort to deal with immediate problems (e.g., groups 
that didn't express themselves, and others that talked about everything but 
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the game; lateness of night |essions that meant cutting short the reflection 
period; initial restlessness that cut short the "learning community" step, 
etc.). The, supervisors, however, would review each session in. the context 
of the particular community, emphasizing that the steps could be followed 
(indeed must be followed) while allowing for individual diversity in each 
group. This was reinforced at the team meeting at the end of October, when 
all the steps were again exhaustively reviewed and put into practice through 
simulations amongst the field 'workers. • ■ • 

1. As the project progressed, the problems encountered in game utili- 
zation were reduced to certain steps within the overall process 
and to certain field workers. ' ■ . 

2. The physicaiymanipulation of the game was rarely a problem; rather, ^ 
we were deanng with deficiencies in the step whereby the game ex- 
perience could be transferred to daily situations. 

3. These were the cases that then received more supervision, with, fre- 
quent field site visits and detailed review of the problem areas. 

•^4. Since absolute standardization, of styles and abilities was never 
contemplated, the differences that appeared were controlled for by 
detailed documentation on a community- by- community basis. 

A secondary control mechanisn?was th-e constant review of the field 
diaries as they were turned in to the office. They were scrutinized for 
both breadth and depth; if questions arose, they would be reviewed with the 
field worker during the next' site visit. o 

In this way, a close touch was maintained with the processes- being fol- 
lowed in each community. Familiarity with the field wo,rkers and the impact 
communities themselves allowed the supervision to be flexible and at the 
same time, it maintained the outlined standards for game utilization.. 
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- jQuality Coatrol of Work Diary 

The oytline of the complete work diary was put together during the 
September field worker training session, keeping in mind in the needs of 
evaluation as such (i.e., massive, detailed, rel i able -information on all as- 
pects ^f the field implementation) and the' physical possibilities of obtain- 
ing such information, given the many demands on the animadores ' time. Major 
emphasis (reflected in the amount of detail requested) was on the implemen- 
tation of the non-formal materials, followed by observations on the communi- 
ty and the animadores' Reactions to it. 

The diary was discussed at length with the field staff; it was made 
clear that in an investigation such as this, the careful documentation of 
he steps followed and results observed are as important as the steps and 
results themselves. At the team meeting in Quito following the first week 
of work in the communities, the diaries of each animador were reviewed and 
discussed, as were the problems encountered in obtaining information and 
strategies for sharpening one's vision. The diaries tended to say too much 
--down to descriptions of breakfast and scenery along the way--or too little; 
the need for focus was obvious. 

In the following weel^, as direct field supervision was stepped up,. the 
field workers could discuss immediate impressions with the supervisors and 
thus gradually adjust their wide-vision experienced to the more narrow needs 
0,^ the evaluation. Another result of the supervision was'to see the need - 
foV) structuring the physical side of the information gathering. The sepa- 
rate forms for game sessions and introduction to the community wene devel- 
oped to pull together these notes into a complete, understandable picture 
of each step. ' " ' 



There was, naturally enough, a great variety of observational 
styles, field experience, and writing skills brought to the evaluation by 
our five to severv member staff; this variety couTd never be eliminated or 
standardized--indeed, it was a key variable. in our evaluation. of the.materi 
als~but rather controlled for through rather rigid formats for the report- 
ing of information, close fieTd supervision, and constant interch^ge of im 
pressions between the field staff and the supe^sors. 
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